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Executive summary 


The project 


Project Based Learning (PBL) is a pedagogical approach that seeks to provide Year 7 pupils with 
independent and group learning skills to meet both the needs of the Year 7 curriculum as well as 
support their learning in future stages of their education. It particularly aims to improve their 
engagement in learning as well as practical literacy skills. This trial evaluates a specific type of PBL 
known as ‘Learning through REAL Projects’, developed by the Innovation Unit—an independent social 
enterprise that aims to improve public sector services. 


A year long pilot project in 2013/2014 with eight schools (763 pupils) established the feasibility of a 
main trial. It provided the opportunity to develop the intervention and test the research procedures. 
The results contained in this report relate to a randomised controlled trial that took place between 
September 2014 and April 2016. Twelve intervention schools (2,101 pupils) and 12 control schools 
(1,973 pupils) were involved in the trial. 


REAL projects was delivered by teachers, supported in many instances by teaching assistants (TAs), 
with further support from senior leadership colleagues. It was delivered for a year in the intervention 
schools with a relatively large proportion of timetabled teaching (varying between 20% and 50%). In 
almost all cases it was delivered to mixed-ability Year 7 classes. 


The intervention used structured cross-subject ‘REAL Projects’ planned by the delivery teachers who 
were supported by Classroom Coaches from the Innovation Unit. Schools were also supported by 
Leadership Coaches. REAL Projects are driven by an essential question which has significant 
educational content. The projects encouraged pupils to create an ‘excellent’ product through drafting 
and redrafting and then to exhibit their work to an ‘authentic’ audience. 


Visits were made to intervention schools at two time points and consisted of lesson observations, 
interviews, and focus groups with school leadership, project leads, class teachers, and pupils. Two 
case studies were also conducted in schools that it was agreed delivered the intervention well. 


Key conclusions 


1. Adopting PBL had no clear impact on either literacy (as measured by the Progress in English 
assessment) or student engagement with school and learning. 


2. The impact evaluation indicated that PBL may have had a negative impact on the literacy 
attainment of pupils entitled to free school meals. However, as no negative impact was found 
for low-attaining pupils, considerable caution should be applied to this finding. 


3. The amount of data lost from the project (schools dropping out and lost to follow-up) 
particularly from the intervention schools, as well as the adoption of PBL or similar approaches 
by a number of control group schools, further limits the strength of any impact finding. 


4. From our observations and feedback from schools, we found that PBL was considered to be 
worthwhile and may enhance pupils’ skills including oracy, communication, teamwork, and 
self-directed study skills. 


5. PBL was generally delivered with fidelity but requires substantial management support and 
organisational change. The Innovation Unit training and support programme for teachers and 
school leadership was found to be effective in supporting this intervention. 
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Security rating awarded as part 


How secure are the findings? of the EEF peer review process 


Overall, the findings have low security. The trial was designed as a two-armed randomised controlled 
trial with schools being allocated to intervention or control groups. It was set up as an efficacy trial 
which aimed to test if the intervention can succeed under ideal conditions. However, 47% of the pupils 
in the intervention and 16% in the control group were not included in the final analysis. Therefore 
there were some potentially important differences in characteristics between the intervention and 
control groups. This undermines the security of the result. The reason that so many pupils from 
schools implementing PBL are missing from the analysis is largely due to five of these schools leaving 
the trial before it finished. Many of the schools in this trial were experiencing challenging conditions, 
and some of the schools that dropped out had a change of leadership team during the trial. It may be 
that successful implementation of the PBL programme is difficult for schools in such circumstances. 
For some of these pupils without Year 7 outcome data, Year 11 data will be available in the future 
from national data sources. This will enable EEF to do a future analysis of the effectiveness of PBL 
which does not suffer from such high rates of data loss. 


What are the findings? 


The trial results did not find that the PBL programme had an impact either on the pupils’ literacy 
performance (as measured by Progress in English 12 tests), engagement, or attendance. The 
analysis did find a statistically significant negative impact on students eligible for free school meals 
(FSM), however no negative impact was found for lower-attaining pupils more generally, which makes 
it difficult to hypothesise why PBL might negatively impact FSM pupils specifically. This adds to the 
uncertainty of the finding. Given the inherent design limitations, high attrition rate, and cross-over in 
terms of group allocation (with some control schools also involved in PBL) we should further limit the 
confidence we have in the above findings as bias in either direction could have been introduced. 


The process evaluation, which was based on classroom observations and feedback from 
headteachers, project leads, teachers, and pupils in the schools, as well as the Innovation Unit 
delivery team, did provide evidence of positive benefits from doing PBL, in particular in terms of 
developing oracy, communication, team working, and research skills. 


The existing international evidence on the effectiveness of PBL is relatively weak, and this research 
contributes to the evidence base from an English perspective. In summary, although PBL is unlikely to 
improve children’s literacy outcomes or engagement, it may enhance the quality of children’s learning, 
particularly improving some of the skills required for future learning and employment. 


How much does it cost? 


The cost per pupil is about £58 per year for the support package from the Innovation Unit. The main 
additional costs are teacher time for training which varied to some degree between schools. The 
Innovation Unit advised a minimum of three days each for three teachers (initial training) plus ongoing 
planning time as necessary including between one and three hours per week for the school PBL team. 


Table 1: Summary of impact on primary outcome 


Effect size i 
Number of Baumer EEF security EEF cost 


(95% confidence months’ 


Seq afore) ky : 
Tay) ave-1)) progress 


eValale) rating 


PBL vs. control 24 -0.13 (-0.31, 0.05) -2 a ELEEELEL 


PBL (FSM-eligible) 


24 -0.24 (-0.41, -0.06) -3 NA ELELEEE 
vs. control 
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Introduction 


Intervention 


The Innovation Unit developed a Project Based Learning (PBL) intervention called ‘Learning through 
REAL Projects’' based on three key principles: 


1. that all pupils are capable of excellence regardless of prior attainment, needs, or 
background; 

2. that student work should matter beyond grades and exams; and 

3. that schools and classrooms are communities of learners and that teachers as well as 
pupils learn with and from each other. 


Broadly speaking, the Learning through REAL Projects intervention involves teachers from different 
subjects designing and planning projects which enable pupils to produce a publicly-exhibited output 
such as a product, publication, or presentation. For example, in all schools the first REAL Project was 
to create and publish an A to Z book. Each school chose their own theme depending on the subjects 
involved (for example, A to Z of technology terms, A to Z of local heroes, A to Z of the school). The 
project involved pupils each researching the topic, writing and redrafting text and images, creating 
proofs, getting publishing quotes, getting a book published, and launching the book at a student-led 
book launch. Other projects varied from school to school and included: 


e pupils learning about the environmental impact of rubbish, creating fashion pieces from 
recycled waste, and putting on a fashion show of these pieces for an audience; 

e learning about local history and creating a museum exhibition within school for local primary 
school children; and 

e ahistory project focusing on slavery and creating and marketing a music album to sell to 
support anti-slavery charities. 


Projects are designed by teachers to be driven by an initial ‘real world’ question so as to allow pupils 
to cover pre-planned significant educational content. Each project leads to a student-created output 
which has a purpose and is displayed to an audience relevant to the project. The focus is on 
producing high quality work to answer the driving question, which has value to the pupils and 
relevance outside of school. Project work involves creating multiple drafts incorporating feedback and 
suggestions (‘critique’) from peers, teachers and others, such as community members, into the work. 
Projects also allow flexibility for student creativity and direction in creating outputs. Projects can still 
involve ‘traditional’ teaching of content where this fits into the knowledge the pupils need to know but 
this is focused towards supporting the project product. 


The REAL Projects intervention provides schools with extensive professional development and 
support for both teachers and leadership staff from experienced teachers and school leaders. A 
similar intervention had been previously used in the U.K. in around eight schools. 


The REAL Projects pedagogy is expected to increase pupils’ deeper engagement with learning and 
with school through the direct relevance of projects to real-life issues or problems, along with the 
linking of different areas of learning (for example, marketing a product involves mathematical ideas, 
design work, communicating ideas, research skills, and literacy). Learning through REAL Projects also 
aims to support pupils in developing skills that are valued by employers in the current market and that 


1 The term ‘REAL projects’ is used interchangeably with ‘Learning through REAL Projects’ and ‘PBL’ (unless 
stated otherwise) and specifically refers to the Innovation Unit version of PBL. 
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will be useful for pupils’ future learning—described by the Innovation Unit as ‘21st Century Skills’. 
These include skills relating to team working, problem solving, communication (including oral 
presentation), leadership, interpersonal relationships, research and analysis, planning, organising, 
and prioritising. Within the pedagogy the development of these skills is assessed and valued 
alongside the academic knowledge learned. While not all skills can be assessed within each project, 
schools were encouraged to focus on one or two ‘21st Century Skills’ to assess alongside relevant 
subject knowledge and the project outcome ‘product’. Although the intervention is a pedagogical 
structure rather than a curriculum or skill-specific intervention, for the purposes of this study, the 
intervention specified that all projects must have a literacy component, primarily aimed at students 
producing high quality written work as part of their final product. It was also recommended that teams 
of teachers delivering the project include an English specialist. 


The Innovation Unit's REAL Projects intervention specifies minimum requirements for schools 
including aspects of organisation, staffing and the curriculum, as well as classroom practices during 
REAL Projects lessons; all of these factors can vary with the existing school practice. The intervention 
involves all pupils in Year 7 in participating schools working in mixed-ability classes. Projects should 
be planned in conjunction with the English department and at least one other discipline so that links 
can be made between traditional instruction and PBL. There is also a requirement for a member of the 
school leadership team to be a part of the team leading the work in the school. 


In addition, Learning through REAL Projects lessons should be timetabled to account for at least 20% 
of the Year 7 timetable (though 40% to 50% was recommended) and at least one project during the 
year should be an extended project accounting for a minimum of six full days learning time. Pupils 
should be taught in blocks lasting at least half a day. Teachers are required to plan projects from 
scratch or to adapt existing projects for their schools, with a recommended one to three hours of 
planning time per week for the team leading the work. This additional planning is required particularly 
as each project has to be planned from start to finish with deadlines for exhibition (for delivery of a 
number of lessons) and delivered in a similar way by different teachers—quite different to traditional 
lessons. Overall, intervention schools did try to meet these requirements although not all schools 
maintained mixed-ability classes throughout the year, and additional planning time specifically for the 
REAL Projects team could not always be provided. Further discussion of implementation can be found 
in the process evaluation section. 


There were also expectations specified of the types of projects to be taught and the essential 
elements of REAL Projects that needed to be evident in those projects. Schools were required to start 
by adapting an existing project set by the Innovation Unit—publishing an A to Z book as described 
above—with the content selected by the school. During the year, schools were expected to adapt an 
additional two existing projects (from a bank of suggested project ideas) as well as create at least one 
original project planned by the teachers that would be teaching it. Each project had to include the 
essential components of: 


e an essential question—driving the whole project; 

e significant academic content built in; 

e multiple drafts of work and critique on work provided by peers, teachers and others; 
e astudent-created final project; 

e public exhibition of the work created; and 

e an authentic audience to view and to comment on the work created. 


Intervention schools were able to incorporate most of these aspects into the projects they planned (in 
particular, multiple drafts and critique were done well in all intervention schools), however the 
‘authentic audience’ was often an issue and the public exhibition was sometimes challenging for 
schools to facilitate. Further discussion of implementation can be found in the process evaluation 
section. 
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Schools allocated to the intervention group were required to pay a contribution of £10,000 towards the 
intervention costs. For this (and additional funding provided by the EEF) schools were provided with: 


e resources and materials—including a school handbook, printed training resources, access to 
online resources, a facilitated online REAL Projects community, as well as webinars 
supporting practice throughout the year; 

e aschool support package (nine days) over the two terms before implementation for 
planning and preparing to deliver REAL Projects that included— 

0 three days of induction (‘Quest’) training (with other schools) that looked at project 
design, teaching, and assessment for the school’s REAL Project team, 

0 three days of individual school support from a Leadership Coach (an experienced 
headteacher with experience of changing school practice) for school senior 
leadership to help with planning and design, 

0 _three days of individual school support from a Classroom Coach (a qualified class 
teacher from an American PBL-based school with extensive experience of teaching 
through Project Based Learning) to support the REAL Projects teaching team with 
planning and project development, and 

0 one day INSET for all staff in the school about Learning through REAL Projects; and 

e anongoing support package for schools during the year of project delivery (27 days) that 
included— 

o 18 days of Classroom Coach time (one visit per fortnight), and 

0 nine days of Leadership Coach time (not always in person—regular telephone calls 
with school leaders were incorporated here along with occasional visits to assess 
progress with the full REAL Projects team). 


The intervention was delivered to pupils during one full school year (September 2014 to July 2015) 
with training and support for schools and teachers starting in February 2014 to allow them to prepare 
for delivery the following school year. 


Learning through REAL Projects is an intervention that involves a high degree of organisational setup, 
intensive initial training, a period of supported planning, and continued intensive support through the 
delivery of the intervention. For many schools it represents a significant change in practice, requiring 
commitment from both school leaders and classroom teachers—hence the level of support provided 
for the delivery of the intervention. 


Background evidence 


A literature review was conducted of the existing evidence for the effectiveness of Project Based 
Learning in primary and secondary schools, and of the factors that can facilitate the implementation of 
PBL. An abridged version is presented here with the fuller literature review included in Appendix B. It 
should be noted that Project Based Learning is different to Problem Based Learning (also sometimes 
shortened to PBL). While there are some similarities, Problem Based Learning concerns 
collaboratively solving an open-ended authentic problem, which is generally derived from professional 
practice, and is often used in medical schools and other post-secondary education. It does not tend to 
focus on creating an end product or exhibition, and due to the nature of the problem there is less 
relevance for an authentic audience. 


Characteristics of Project Based Learning 


Project based learning is a student-centred form of instruction based on three constructivist principles: 
learning is context-specific, learners are involved actively in the learning process, and they achieve 
their goals through social interactions and the sharing of knowledge and understanding (Cocco, 
2006). It is considered to be a particular type of inquiry-based learning where the context of learning is 
provided through authentic questions and problems within real-world practices (Al-Balushi and Al- 
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Aamri, 2014) that lead to meaningful learning experiences (Wurdinger, Haar, Hugg and Bezon, 2007). 
The uniqueness of PBL is the construction of an end product, a ‘concrete artefact’ (Helle et a/., 2006) 
which represents pupils’ new understandings, knowledge, and attitudes regarding the issue under 
investigation—often presented using videos, photographs, sketches, reports, models, and other 
collected artefacts (Holubova, 2008). 


It is argued that the freedom and challenge that pupils experience as a result of solving the problems 
that arise in designing and building their projects result in high levels of student engagement 
(Wurdinger et al., 2007). It is also said to foster self-regulated learning and promote pupils’ conceptual 
knowledge within a systematic process of documenting and reflecting on learning (Barak, 2012). 
Within project based learning, pupils learn to be self-reliant through goal-setting, planning, and 
organisation; they develop collaboration skills through social learning; they also become intrinsically 
motivated by being encouraged to exercise an element of choice while learning at their own level 
(Bell, 2010). 


Existing evidence 


The existing evidence for a causal link between PBL and attainment outcomes seems to be weak. 
Most of the reviewed studies did not involve random allocation of participants to control and 
experimental groups and, as a result, a causal link between project based learning instruction and 
positive student outcomes has not been established. The majority of studies were based on a quasi- 
experimental pre-test/post-test design with some baseline equivalence established for the outcomes 
measured at the classroom level. Some studies with a less robust design for inferring causality were 
based on observations of pupils’ behaviour, attitudes, and accomplishments in a project based 
learning environment without the presence of a comparator group (for example, Barak and Asad, 
2012; ChanLin, 2008; Cuevas, Lee, Hart and Deaktor, 2005). Other studies have used state 
standardised test averages against which to compare the performance of pupils (Geier et a/., 2008). 


Looking specifically at studies at secondary school level, a number of non-randomised studies 
demonstrated positive effects on content-specific learning through use of a project based learning 
curriculum—although these have usually been small pieces of research on short specific projects with 
very few U.K.-based studies. However, there appears to be no previous research which has 
established the effectiveness of PBL on literacy measures specifically; most papers focus on science, 
technology and history attainment outcomes. 


Positive results have also been reported with primary age pupils although again most of the studies 
have been conducted in non-U.K. contexts and are small-scale pieces of research (for more detail 
regarding research findings in primary schools, see Appendix B). 


In Oman, Al-Balushi and Al-Aamri (2014) conducted a small scale, two-class, quasi-experimental 
study with 62 11th-grade female pupils (equivalent to Year 12 in the U.K.) that explored the effect of 
environmental science projects on pupils’ environmental knowledge and attitudes towards science. 
The findings were positive with the experimental group significantly outperforming the control group in 
an Environmental Knowledge Test and the Science Attitudes Survey. In history learning, Hernandez- 
Ramos and De La Paz (2009) had eighth-grade pupils in the U.S. (equivalent to Year 9 in the U.K.) 
learn to create multimedia mini-documentaries in a six-week history unit. Compared to pupils who 
received traditional instruction (in a pre-test/post-test study), pupils that engaged in the project based 
learning curriculum demonstrated positive affective benefits and significant gains in content 
knowledge as well as historical thinking skills. However, again this was a weak study with only two 
classes with two different teachers which were not randomly allocated. Other quasi-experimental 
studies conducted in the U.S. (Hsu, Van Dyke, Chen and Smith, 2015; Geier et a/., 2008) looking at 
science education reported positive effects for the project based learning environment compared to a 
comparison group in teaching seventh- and eighth-grade pupils (Years 8 and 9 in the U.K.). They 
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found better development of argumentation skills and science knowledge and understanding, better 
process skills, and higher exam pass rates on a state-wide test. 


Other small studies in Israel (Barak and Asad, 2012) and Taiwan (Lou, Liu, Shih and Tseng, 2011; 
ChanLin, 2008)—again focusing on project based learning in science—have indicated that a PBL 
environment led to higher learner motivation and increased interest in learning science, as well as 
increased enjoyment and engagement in learning STEM subjects. In ChanLin’s (2008) qualitative 
study, PBL supported pupils in Taiwan in synthesising and elaborating knowledge in science 
exploration tasks. 


Project based learning has been explored as a method of instruction with low-achieving pupils in 
Israel (Doppelt, 2003) and the U.S. (Cuevas et a/., 2005), and with ‘second chance’ school pupils in 
Greece (Koutrouba and Karageorgou, 2013) with positive outcomes. Doppelt (2003) found—through a 
field research project with 54 fifteen- and sixteen-year-olds—that scientific-technological project based 
learning helped improve low-achieving pupils’ motivation and self-image by allowing them to succeed 
early on in the process; this led to more pupils achieving college admittance requirements. 


Some international studies have, however, shown mixed results. For example, in a quasi-experimental 
study with 13-year-old children taking computer courses in Greece, Boubouka and Papanikolaou 
(2013) found no significant effect of project based learning on student achievement but a statistically 
significant positive effect on self-perceived learning performances. 


In the U.K. context, Boaler (1998) conducted a longitudinal study of mathematics instruction 
comparing an open, project-based environment to a traditional approach following two cohorts of 
pupils in two British secondary schools from Year 9 to Year 11. This study employed a closely- 
matched control group in terms of socioeconomic status, prior mathematics instruction, and 
attainment. A variety of instruments were used to measure pupils’ skills, attitudes, and attainment. The 
main finding was that the two groups developed different forms of knowledge. The pupils learning 
mathematics in the project-based environment developed conceptual understanding which often 
required creative and deeper thinking in contrast to the procedural knowledge acquired by the 
traditional instruction group which was mainly based on information recall. In addition, more pupils at 
the project-based school succeeded in passing the General Certificate of Secondary Education 
(GCSE) at the end of the three-year study than those pupils receiving the traditional instruction. 


Because of the transitional nature of pupils in Year 7 in this project, it is also relevant to look at some 
of the existing evidence for pupils in later primary school. Kaldi, Filappatou and Govaris (2011) 
conducted a quasi-experimental study in primary schools and from this argued that primary-age pupils 
can develop content knowledge and group work skills in addition to motivation and positive attitudes 
towards peers from a different ethnic background through project based learning instruction. Similarly, 
Karagalli and Korur (2014) conducted a quasi-experimental study in Turkey with fourth-grade science 
pupils (equivalent to Year 5 in the U.K.) and found a statistically significant effect in terms of academic 
achievement and retention of knowledge for the PBL pupils. However, like many of the other studies 
included, this had a very weak design involving only pupils within one class volunteering to be part of 
the treatment, and had flaws to the analysis conducted. 


Recommendations for implementing PBL 


The literature was reviewed for the factors that facilitate the use of project based learning and how 
teachers can best support it. Six recommendations came from this search of the literature: 


1. Student support: pupils need to be effectively guided and supported through the PBL 
process; emphasis should be given on effective time management and student self- 
management including making safe and productive use of technological resources. 


Education Endowment Foundation 10 


Project Based Learning 


2. Teacher support: regular support needs to be offered to teachers through regular 
networking and professional development opportunities. Support from the school senior 
management is crucial. 

3. Effective group work: high quality group work will help ensure that pupils share equal 
levels of agency and participation. 

4. Balancing didactic instruction with independent inquiry will ensure that pupils develop a 
certain level of knowledge and skills allowing them to comfortably engage in independent 
work. 

5. Assessment emphasis on reflection, self, and peer evaluation: evidence of progress 
needs to be regularly monitored and recorded. 

6. An element of student choice and autonomy throughout the PBL process will help pupils 
develop a sense of ownership and control over their learning. 


The Learning through REAL Projects approach incorporates all these recommendations within the 
programme to some degree. There is particular focus on providing teachers with regular support 
throughout the year and on facilitating the relationship between senior management and REAL 
Projects teachers. Strategies (including suggested activities) for supporting pupils with becoming 
familiar with the PBL process and with managing group work are provided through the online 
community and by the Classroom Coaches. The programme also gives guidance and suggestions for 
assessing work and monitoring progress with suggested pro formas. 


Rationale for the Current Project 


Project Based Learning and similar approaches (such as the RSA’s Open Minds project) have been 
advocated, developed to various degrees, and used in school teaching for many years. Currently, a 
few schools in the U.K. are using PBL as the basis of their whole curriculum including some academy 
schools and free schools from primary through to sixth-form level. A larger number of other schools 
are using the PBL approach for specific subjects, year groups, or for off-curriculum time, some of 
which have been using this approach for a number of years. PBL seeks to offer a way to engage 
pupils more deeply with learning, to interest them in areas they might not have thought about and to 
develop their skills for life beyond school. However, as demonstrated by the above review, very few 
impact evaluations have been carried out, particularly within the U.K. education system. This project 
therefore provides a timely opportunity to assess both the impact of a PBL intervention and its 
potential further adoption in schools. 


Due to the complex nature of REAL Project and its implementation, it was decided by the EEF, 
Innovation Unit, and the evaluators to include a pilot year where the intervention would be piloted in a 
small number of schools and further developed. One of the aims of the pilot year was for the 
developers to decide on the right balance between support and cost. The Innovation Unit used 
feedback from schools and the research team to better refine and define the intervention. These 
changes were made prior to the full trial. 


The Learning through REAL Projects intervention had not previously been tested in a research study, 
however the previous evidence indicated that the PBL approach showed promise. Therefore an 
efficacy trial testing the intervention when delivered in ideal conditions (with a high level of support 
provided by expert classroom teachers from an American PBL school) was chosen for this trial. 


As well as exploring the impact of the intervention on attainment and engagement, this evaluation 
aimed to investigate the facilitators and barriers to English schools implementing and maintaining this 
intervention over a two-year period. The results of this project aimed to inform the feasibility of using 
the approach in schools, particularly within the Key Stage 3 curriculum, and suggest methods that 
could allow the intervention to be scaled up for more wider delivery if successful. 
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Evaluation objectives 
Pilot development year questions 


1. What is the existing evidence underpinning the PBL pedagogy? 

2. Howcan the Learning through REAL Projects intervention be developed ready for a 
randomised controlled trial? 

3. Do the planned trial research methods work when piloted? 


Impact evaluation questions 


1. What is the impact of the Learning through REAL Projects intervention on literacy when 
comparing pupils in intervention schools with those in control schools? 

2. What is the impact of the Learning through REAL Projects intervention on engagement 
with school and learning for the participating pupils, compared to those in control 
schools? 

3. Are there differential impacts of the Learning through REAL Projects intervention on 
different subgroups of pupils such as groups based on FSM status, prior attainment, and 
gender? 


Process evaluation questions 


1. What is the feedback from pupils on the Learning through REAL Projects intervention? 
2. What is the feedback and professional assessment of the Learning through REAL 
Projects intervention from teachers and school leaders? 

How effective and appropriate is the training and professional development provided? 
What are the commitments and implications for schools adopting the programme? 
What are the direct and indirect costs of delivering the programme? 

What are the key success factors and barriers to successful implementation? 

Are there any other areas of the programme that could be further developed (following 
the completion of the trial)? 


Or Ol Pe 


Link to protocol: 


https://educationendowmentfoundation.org.uk/public/files/Projects/EEF_Project_Protocol_Proj 
ectBasedLearning.pdf 


Ethical review 


Ethical approval for the evaluation was obtained from the Board of Ethics in the School of Education, 
Durham University on 20 November 2012. This was updated on 25 March 2014 to accommodate 
changes to the research design. See Methodological section below for details of these changes. 


Consent was given by the headteacher of each school involved to take part in the research. Control 
schools signed a memorandum of understanding with the Innovation Unit and intervention schools 
signed a services agreement agreeing to the requirements of the trial. Additional information sheets 
about the evaluation were also sent to all schools detailing the specifics of the research and how 
schools’ data would be used. See Appendix C for examples of the school agreements, information 
sheets, and consent forms. 


The intervention was delivered to all pupils as part of the curriculum but parents were given the 
opportunity to opt-out of their child’s inclusion in the research through letters provided by the research 
team and sent by the schools to parents (also included in Appendix C). 
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Assessment at the end of the project was conducted across the full year groups; where parental opt- 
out had been received the papers were not collected or marked. One school required collecting opt-in 
consent to undertake the testing. In this case, as so few opt-in responses were received (30 out of 
125), it was deemed unethical to run the assessment as the sample would be too small to be able to 
generalise results from this school. 


Pupil focus groups were carried out in some schools as part of the process evaluation. Opt-in consent 
was obtained for these with information and consent forms sent home to parents and pupils assenting 
at the beginning of the focus groups. Teacher, headteacher, and developer interviews and focus 
groups were also conducted and oral consent was obtained to conduct these and to record them for 
research purposes before the interviews. 


Project team 


Innovation Unit (Developer)—responsible for programme development, recruitment to the 
project, and delivery of intervention: 


Programme Lead: Louise Thomas. 

Operations Manager: Kim Schilling. 

Leadership Coaches: David Jackson, Chris Cotton, Angela Armytage, Jan McKenley-Simpson. 
Classroom Coaches: John Bosselman, Cara Littlefield, Loni Bergqvist, Kay Flewelling. 


Durham University (Evaluator)—overall responsibility for delivering the evaluation, including 
trial conduct, assessment collection, and process evaluation: 


Principal Investigator: Dr Andy Wiggins. 

Lead Researcher: Victoria Menzies. 

Researcher (Literature review and process evaluation): Dr Dimitra Kokotsaki. 
Advisor (Pedagogy and Literature review): Professor Joe Elliott. 

Advisor (Assessment): Dr Christine Merrell. 

Research Administrator: Clare Collyer. 


York Trials Unit, University of York (Evaluator)—responsible for contributing to research 
design, conducting randomisation and analysis: 


York project lead: Professor David Torgerson. 
Lead Statistician: Professor Catherine Hewitt. 


Trial registration 


The trial was registered on the ISRCTN registry, study ID ISRCTN38596728. 
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Methods 


Trial design 


The evaluation project took place in two stages: an intervention development and research pilot stage 
during the first year of the project and a full randomised controlled trial following this. 


Pilot/development phase 


During the pilot/development stage, the intervention and support to be provided to schools was 
developed, refined and fully described by the Innovation Unit. This development was based on how 
the eight pilot schools (recruited by the Innovation Unit and having a range of previous experience of 
using PBL) used the programme, as well as independent feedback from the evaluation team about 
pedagogy, delivery, organisation, and support. During this period the evaluation team: 


e conducted a literature review of the previous evidence for the effectiveness of Project Based 
Learning and the aspects that facilitate its delivery in schools; 

e observed the initial three-day training programme and ongoing events provided for pilot 
schools; 

e conducted phone interviews with the PBL lead in seven of the eight pilot schools; 

e made visits to the three pilot schools most likely to be generalizable to the general population 
to observe PBL lessons, conduct interviews (with senior management and delivery staff), and 
to conduct a focus group with pupils; and 

e provided feedback to the Innovation Unit on the delivery of the intervention (throughout the 
year with final date for feedback in June 2014). 


During this stage of the project, the delivery of the primary outcome measure and newly developed 
engagement survey were piloted and feedback gathered from schools on the acceptability of the 
research processes. Interview and focus group schedules were also developed, piloted, and refined 
for the main project. This research piloting led to changes in the evaluation protocol for the main trial 
including an updated description of the intervention, and changes to secondary outcome measures 
and methods of delivering the assessment. The original protocol only included the academic outcome 
measures of literacy, maths and science. The piloting revealed that schools were not including maths 
as part of project based learning and that science was included only in one school. It was therefore 
decided to retain the literacy focus but to introduce an ‘engagement’ secondary outcome as this 
seemed more aligned with the intervention’s aims. The original assessment delivery method was via a 
computerised assessment. The piloting also revealed that most schools did not have adequate 
facilities to deliver an IT assessment via computers for the full year group. 


All aspects of this report hereafter refer to the Main Trial. 
Main Trial 


The main trial was a cluster two-armed randomised controlled trial with randomisation to the Learning 
through REAL Projects intervention group or to a ‘business as usual’ control group conducted at 
school level. The trial took place over the course of one year, although process evaluation data was 
collected for an additional year. 


The original protocol stated that this would be a two-year trial with outcomes collected at the end of 
the second year. However, a decision was made by the EEF and the Innovation Unit not to fund the 
intervention for the second year as this would have required additional funding beyond the initial grant. 
It had been expected that the Innovation Unit might have been able to obtain additional funding for 
this second year, however this was not possible. The trial was therefore shortened to one year in a 
revised protocol agreed in January 2015 with outcomes collected after one year of delivery. Given the 
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organisational adjustments, staff commitments, and professional development involved for schools in 
delivering the intervention, a two-year trial would have allowed better embedding of REAL Projects 
and enabled schools to have become more independent with their delivery. It is also reasonable to 
assume that measuring outcomes after only one year might underestimate the effect of the 
intervention, however it was also necessary to take into account the additional cost to schools and the 
EEF of delivering the intervention. 


Schools in the intervention group were asked to deliver the intervention throughout the year of the 
project, attend the required training, and comply with the minimum requirements as described above 
in the intervention section. Schools allocated to the intervention group were required to pay the 
Innovation Unit £10,000 towards the support and materials provided. 


Schools in the control group were asked to continue with their usual practices of teaching at Year 7— 
‘business as usual’. No incentives were provided other than the option of purchasing the intervention 
for delivery after the project. Control schools were not given access to intervention materials (although 
limited materials were available on the Innovation Unit website) and were given no access to the 
support and professional development provided by the Innovation Unit between the start of the project 
and February/March 2015. From this time, if desired, control schools were able to pay for and access 
a reduced form of Learning through REAL Projects training and resources in order to plan for delivery 
in the 2015/2016 school year. Although this training took place before the post-testing in June and 
July 2015, control schools were committed not to implement REAL Projects until September 2015. 
The process evaluation revealed, however, that some control schools were doing a version of project 
based learning during the trial although this was without the direct support of the Innovation Unit and 
their coaches. 


This two-armed cluster RCT design was chosen in order to provide robust estimates of the impact of 
the intervention. Although the trial was underpowered with only 24 schools, it was expected that the 
trial would give an indication of the potential impact and would also serve as a study of the feasibility 
of conducting an RCT of an intervention requiring substantial organisational change across schools. 


Participant selection 


Schools were eligible to participate in the project if they were a mainstream state secondary school in 
any area of England willing to be randomised to implementing the intervention (and pay a contribution 
of £10,000) or to be part of a business as usual control group. Schools had to be willing to undertake 
all research related procedures and to have the capacity to deliver the intervention (as assessed by 
the Innovation Unit). All pupils in Year 7 in participating schools were eligible to take part and schools 
were asked to include all Year 7 pupils. 


Twenty-four schools were recruited by the Innovation Unit using their own networks of schools; 
recruitment also took place through a national media campaign with articles about the project in the 
TES magazine. Schools were asked to provide initial information on a registration form and were then 
contacted by the Innovation Unit. Each interested school was visited by a member of staff from the 
Innovation Unit in order to assess the school’s capacity to deliver the intervention (for example, 
whether they would be able to make the required timetable changes and provide the required 
staffing). A recruitment conference was also held for interested schools to give details of what the 
intervention and research would involve. This was hosted by the Innovation Unit and the Evaluation 
Team. 


Initially the Innovation Unit provided details of twenty-four schools for randomisation, however shortly 
after providing this list, and before randomisation, it informed the evaluation team that one school had 
withdrawn. This school was then replaced and randomisation took place in January 2014 with schools 
informed of their allocation shortly after. While schools had consented to be randomised, they did not 
sign a full agreement until after the randomisation had taken place. This was due to the nature of the 
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agreements being legally different, particularly with regard to the cost the intervention group was 
required to pay. Consent was given by the headteacher of each school to take part. Control schools 
signed a memorandum of understanding with the Innovation Unit and intervention schools signed a 
services agreement agreeing to the requirements of the trial and delivery of the intervention. An 
evaluation information sheet was provided to all schools separately to this. Parent opt-out consent 
was not sought until after the 2014/2015 school year had started as all pupils were joining a new 
school at that point. Opt-in consent was sought from parents for process evaluation student focus 
groups, as relevant, before these took place. Examples of the school information sheet, parental opt- 
out consent forms, and parental opt-in consent forms for student focus groups are included in 
Appendix C. 


Outcomes measures 
Primary outcome: literacy 


The primary outcome was literacy attainment, as collected through the Progress in English 12 
assessment (GL Assessment, 1994, henceforth ‘PiE’). Literacy was chosen as the primary outcome as 
the intervention has an explicit focus on literacy and requires the involvement of staff from the English 
department. Literacy is also particularly important as a progress measure to schools, especially at Key 
Stage 3, so it was important for the trial to investigate whether Learning through REAL Projects could 
be detrimental to pupils’ literacy achievement. 


The short form of the PiE 12 assessment was used as it is a standardised English and literacy 
assessment often used for summative assessment in school. It includes two reading comprehension 
passages (fiction and non-fiction) along with spelling and grammar exercises. It does not include any 
extended writing tasks. The overall raw score from the PiE 12 assessment was used in the primary 
outcome analysis. 


Data from Key Stage 2 (KS2) SATS was obtained from the National Pupil Database (NPD) to use as a 
baseline assessment. Fine grade reading scores were chosen as the baseline outcome to use in the 
analyses as the correlation between Reading and PiE 12 was expected to be greater than other KS2 
outcomes, giving the trial more power to detect an effect of the intervention should one exist. 


Secondary outcomes: engagement 


Secondary outcomes aimed to capture how actively engaged pupils were with school and learning. 
Engagement with school and learning was defined as feelings about, disposition to, and participation 
in, school, school life, and learning activities. These outcomes were chosen because a primary aim of 
the intervention was to increase pupils’ engagement with the learning process and to give pupils the 
skills and motivation to continue learning beyond the classroom. 


Measurement of engagement was done in two ways: through an ‘Attitudes to School and Learning’ 
survey that was developed to investigate pupils’ self-reported engagement with school, and 
additionally through school attendance record data collected from the NPD. Collecting both survey 
data and attendance data therefore provided a way to investigate two aspects of this and to 
triangulate findings. 


Attitudes to School and Learning survey 


The ‘Attitudes to School and Learning’ survey was developed by the Durham University team to cover 
a broad concept of school engagement and learning in the classroom and beyond. The development 
of this measure began with a literature review of how ‘engagement’ was conceptualised and to review 
existing measures of engagement which used self-report data. The literature suggested engagement 
should be looked at in terms of behavioural/operative/doing elements, affective/emotional/feeling 
elements, and cognitive/intellectual/thinking elements. 
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Validated items from existing measures (which could be sourced), covering the above aspects of 
engagement, were collated into a database and categorised by engagement concept or theme. From 
these, the themes most relevant to the aims of the Learning through REAL Projects intervention were 
selected; items from these themes were then adapted to suit the U.K. context and, where necessary, 
made age-appropriate. The draft measure and items were shared and discussed with the Innovation 
Unit and their feedback incorporated. 


For most items, student responses were given on a four point Likert scale to indicate how true the 
statement is. Items were not specific to individual lessons but to school and classes generally. The 
initial measure for piloting included sixty-three items looking at: 


e identity with school; 

e relationships with teachers and other pupils; 

e motivation for learning; 

e self-efficacy at school; 

e involvement in the classroom; 

e student learning behaviours (cognitive and metacognitive); and 
e teacher behaviours in class. 


The measure was piloted with 356 pupils in six schools. The data was subjected to factor analysis and 
scale reliability analyses. Following this, we removed items that did not work as intended (for example, 
that were not appropriate, did not fit with the scale, or reduced the reliability of the scale). The final 
attitudes measure used in the main trial (included in Appendix D) contained 53 items that principal 
components analysis revealed included seven subscales: 


e active interest and enthusiasm; 

e positive pupil ethos; 

e pupil self-efficacy; 

e motivation for learning; 

e = effort and behavioural engagement; 
e attitudes towards teachers; and 

e pupil agency in the classroom. 


Cronbach’s alpha of 0.95 was calculated for this measure, indicating a high level of internal 
consistency and therefore good reliability. 


The total score on this measure, named the ‘Attitudes to School and Learning questionnaire’, was 
used as the secondary outcome. 


School attendance data 


Attendance at school was also used as a proxy measure for engagement. It was expected that pupils 
who were more engaged with school would have fewer absences than those who were less engaged. 
Data for this outcome was requested from the NPD. A percentage attendance score was calculated 
using the ‘number of sessions possible’ field and the ‘number of sessions attended’ field using data for 
the first two school terms of Year 7. 


Outcome measure collection 


Outcome measures were collected in June and July 2015 after schools had been implementing the 
intervention for three terms (with training taking place for teachers to prepare for delivery for two terms 
previously). The assessment and survey were delivered in one session lasting up to one and half 
hours under exam conditions and in most cases invigilated by a researcher independent of the 
schools and blind to the allocation of the school (and supported by school staff). In one case the 
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school administered the assessment with their own invigilators without supervision from the evaluation 
team due to difficulties with scheduling, and in one case a member of the research team not blinded to 
the school allocation administered the assessment. In two schools it was not possible to fit all pupils 
into one hall and the independent invigilator only assessed half the pupils while school staff assessed 
the remaining pupils. After sitting the assessments, the papers were sent directly to, and were marked 
by, GL Assessments and the data shared with the research team. The surveys were returned directly 
to the research team by the external invigilator (or in one case, by the school). The research team 
also returned the individualised results of the assessment and summary data of the attitudes survey to 
schools over the summer. 


Changes from original protocol 


In the original protocol, outcomes in literacy, maths, and science were expected to be collected using 
the CEM INSIGHT system. Due to the trial being shortened by a year, the INSIGHT system 
assessment was no longer appropriate for the age group and a decision to use the PiE measure was 
made instead. Maths and science outcomes were dropped from the revised protocol at this stage as 
the pilot project had revealed that schools were not consistently including maths or science in their 
REAL Projects. Feedback from schools about the time demands of additional assessments was also a 
factor in this decision, as it was in the decision not to also ask pupils to complete an additional 
measure of their future aspirations. 


Sample size 


The sample size was determined by the number of schools that the Innovation Unit could support with 
the funding they received from the EEF. This allowed for 12 intervention schools and a further 12 
control schools which, assuming 125 pupils per year-group, gives a sample size of 3,000 pupils. We 
calculated the minimum effect size that a trial of this sample size would be able to detect by assuming 
a pre- and post-test correlation of 0.70 and an intra-cluster correlation coefficient of 0.19. The design 
effect would be 24.56, giving an effective sample size of 122. This is increased by the pre- and post- 
test correlation to give an ‘effective’ sample size of 240. This would allow us to detect, with 80% 
power, 5% two sided alpha and 20% attrition, an effect size of 0.40 (p = 0.05). 


Randomisation 


The Innovation Unit provided the Durham evaluation team with a list of schools that they had signed 
up to the project and information on the percentage of FSM students in each school and the size of 
the Year 7 group. Schools were coded, and names removed, and the list passed on to the York Trials 
Unit that developed the randomisation procedure and allocated schools. Schools involved in the pilot 
were not eligible to take part in the main trial to avoid contamination effects. 


Once pupil baseline data was received, schools were allocated on a 1:1 basis to either receive the 
intervention for their incoming Year 7 cohort in 2014/2015 (the intervention school group) or to 
continue with ‘business as usual’ with the Year 7 cohort (the control group). The allocation was 
undertaken by an independent statistician using minimisation implemented in minimPy 0.3 (Saghaei 
and Saghaei, 2011). The independent statistician was not involved in the recruitment of schools and 
pupils, ensuring independent randomisation. 


We chose to use minimisation, rather than simple randomisation or stratified randomisation using 
blocks, because this method of allocation allowed a better balance in terms of observable school-level 
characteristics compared with other allocation methods. This improves the credibility and statistical 
efficiency of a trial when the number of schools allocated is less than 100. The minimisation factors 
included the size of the Year 7 intake, the percentage of FSM pupils, whether the school was a free 
school or not, and whether the school stated on its website that it was already using PBL methods. 
Size of school seemed likely to be a factor affecting schools’ ability to make teaching and timetable 
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changes. Similarly, free school status was included in the randomisation due to the perception that it 
would be easier to make timetable and teaching changes in a free school than in existing non-free 
school secondary schools (as seen in the pilot year). The IU was consulted about whether schools 
already using PBL methods (according to their websites) should be included in the sample: the IU was 
satisfied that no school was implementing the same model of PBL and it was therefore decided to 
include these schools in the sample and to control for this in the randomisation. 


Naive minimisation with base probability 1.0 (deterministic minimisation) was conducted using the 
following factors: 


e number of pupils in Year 7 (two levels: less than 180 pupils and more than 180 pupils); 

e percentage of pupils eligible for FSM (two levels: less than 36% and more than 36%); 

e whether the school was a free school or not; and 

e whether the school stated on its website or prospectus that it was using a form of Project 
Based Learning. 


The variables ‘percentage of free school meals’ and ‘size of the Year 7 intake’ were continuous and 
therefore needed categorising. Cut-off values for levels were chosen based on baseline summary 
statistics. It was decided to split the two variables into two groups based on the median value. This 
was 36 for FSM proportion and 180 for size of Year 7 intakes. 


Naive minimisation was deemed to be sufficient as the allocations were conducted in a batch, rather 
than prospectively, meaning predictability was not a concern and hence a random element was not 
required. The school data was anonymised when received. The 24 schools were labelled a—x. Before 
being entered into the minimisation programme, the schools were randomly ordered in Stata v13. A 
random number from a Uniform [0,1] distribution was generated for each school using the seed 
1556347. This variable was sorted in ascending order, and a consecutively numbered ID generated 
for the schools. 


Analysis 


Analyses were conducted in Stata® version 13 (Stata Corporation, College Station, Texas) using the 
principles of intention-to-treat, meaning that all schools and pupils were analysed in the group to 
which they were randomised, irrespective of whether or not they actually received the intervention 
throughout. Statistical significance was assessed at the 5% level unless otherwise stated; 95% 
confidence intervals are provided as appropriate. Model diagnostics were used to check model 
assumptions and transformations considered if assumptions did not hold. 


Effect sizes are presented relating to analyses alongside 95% confidence intervals. In this report, 
effect size is defined as: 


A= B intervention 


yo, +d; 


where [intervention is the difference in mean scores between the intervention and control groups 
adjusting for KS2 reading fine grade scores and minimisation factors, o< is the residual standard 
deviation, and os is the standard deviation between schools. 


Baseline data 


School and pupil characteristics are presented by group to assess balance, however no formal 
statistical testing was conducted. As the randomisation process has been performed properly in this 
trial we know that the null hypothesis of no difference between treatment groups will be true. Hence 
any differences between groups that may be present would be due to chance alone. Undertaking 
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baseline testing would be uninformative and potentially misleading (Altman and Dore, 1990; Begg, 
1990; Senn, 1994). 


Descriptive analyses 


Raw unadjusted outcome results are summarised by group. An estimate of the intra-cluster correlation 
coefficient (ICC) associated with school for the primary outcome of PiE is presented alongside a 95% 
Cl. The correlation between the primary outcome of PiE scores and KS2 reading fine grade level was 
also estimated. 


Primary analysis 


The primary objective of this study was to investigate the impact of the intervention on literacy for the 
target pupils in the intervention schools compared to the control schools. The difference in literacy 
attainment between pupils in the intervention group and those in the control group was compared 
using a linear mixed model with PiE score as the response variable. Group allocation, KS2 reading 
level fine grading, FSM status, number of Year 7 pupils, whether free school or not, and currently 
advertised PBL (yes or no) were collected at baseline and were included as covariates in the model. 
The primary analysis needs to reflect the restrictions implied by the minimisation, hence FSM status, 
number of Year 7 pupils, whether free school or not, and currently advertised PBL (yes or no) were 
included as covariates in the model irrespective of their prognostic value (Committee for Medicinal 
Products for Human Use (CHMP), 2015). Adjustment was made for cluster randomisation through the 
inclusion of school as a random effect. 


Secondary analyses 


A repetition of the primary analysis with an additional adjustment for gender was included in the 
model. This was the only additional factor pre-specified in the protocol that was not included as a 
minimisation factor and thus in line with the EEF statistical guidance. An analogous approach to the 
primary analysis was also used to assess for difference between the intervention and control pupils in 
terms of the secondary outcomes of the attitudinal survey total scores and attendance data; the same 
adjustments as the primary analysis were undertaken. Model assumptions were checked for each 
analysis by visual inspection of normal plots to assess deviations from normality and plots of residuals 
against predicted values to assess uniform variance. The underlying assumptions of normality for the 
analysis were in doubt for attendance data with the residuals showing clear skew as the points 
deviated from the central line. When the assumption of normality is in question, then we can often 
transform the data to another scale where the assumption of normality is reasonable. A transformation 
applies the same mathematical function (here arcsine) to each data point. Hence the outcome was 
transformed prior to analysis. Given the skew of the data and type of outcome data, an arcsine 
transformation was utilised. 


Subgroup analysis 


Three pre-specified subgroup analyses (FSM status, gender, and prior attainment) were explored. 
The effect of the intervention on pupils on each of the predefined variables was assessed by including 
an interaction term between each baseline variable of interest and group allocation in a repetition of 
the primary analysis. An additional analysis repeated the primary analysis for the subgroup of pupils 
eligible for FSM to be in line with the funder’s requirements. At the request of a reviewer, a further 
exploratory subgroup analysis (FSM status) was also undertaken for the secondary outcome of 
attitudinal total scores. 


Missing data 


The impact of missing data on the primary analysis was assessed using multiple imputation by 
chained equations with 20 imputations. Missing values were replaced by plausible substitutes based 
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on the distribution of observed data (rather than a single estimate such as the mean) also including 
randomness to reflect uncertainty; this was repeated 20 times to create 20 datasets which were 
combined to give estimates and 95% confidence intervals (Royston and White, 2011; Azur, 2011). 
Baseline characteristics and PiE scores were both imputed using the same covariates (PIE12, KS2 
reading, minimisation factors, and group). 


Implementation and process evaluation 


Due to the multifaceted and complex nature of the intervention and the implications of this for schools 
in terms of adopting PBL, the process evaluation was quite extensive. It included: 


e observations of initial training for schools; 

e cross-sectional visits to intervention schools collecting observation and interview/focus group 
data from school leadership, class teachers, and pupils; 

e interviews with control schools about their teaching and PD practice; 

e separate focus groups with Leadership Coaches and Classroom Coaches; 

e interviews with the Innovation Unit programme leaders; 

e intervention teacher survey at the end of the intervention year: and 

e case studies of how two schools used the Learning through REAL Projects intervention. 


The methods for each aspect are detailed below. 
Observations of initial training for schools 


At least one researcher attended a day of training for headteachers and REAL Project Leads in April 
2014 and two of the three days of Quest training for the teaching team that would be involved in 
delivering the intervention in May 2014. These sessions were for all intervention schools together and 
took place in London. Observations were conducted to familiarise the evaluation team with the 
requirements on schools of delivering the intervention and to allow the team to understand how 
training had been delivered to schools so as to explore this as a topic in interviews. Open notes were 
taken during these observations and informal conversations were held during these days with those 
participating and those leading the training. 


Cross-sectional visits to intervention schools collecting observation and interview/focus group 
data from school leadership, class teachers and pupils 


Six out of the twelve intervention schools were chosen at random for two school visits. At the time of 
arranging visits, one of the six pulled out from the project and from delivering the intervention and was 
unwilling for a visit to be made. Five schools therefore remained in the sample. Structured visits were 
made by two researchers to all five schools between April 2015 and June 2015. Visits, where 
possible, included observations of REAL Projects lessons (using an observation schedule with 
headings and free form responses), an interview with the headteacher (or nominated senior leader), 
an interview with the Learning through REAL Projects lead, a focus group involving three to five class 
teachers delivering the REAL Projects lessons, and a focus group with a group of pupils in the school. 
All interviews and focus groups used a semi-structured schedule covering themes (depending on the 
relevance to participants) including: organisation of REAL Projects, staffing, training and support, 
project delivery and fidelity, impact, and barriers and facilitators to delivery. 


s conducted. 


Table 2 below shows which of these aspects were conducted in each school and the total number of 
observations, interviews, and focus groups conducted. 
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Table 2: Number of interviews and observations across process evaluation schools 
(conducted during first year/conducted in second year) 


Case 


Pupil 


end Headteacher Other PBL Teacherls, focge Lesson 
y SLT lead other staff fo} o-Y-) aVecta Loy a} 
Ted aTore) | group 
School A 1/0 0/1 1/1 1/1 1/0 1/0 
School B 1/0 1/1 1/1 1/1 1/0 1/0 
School C 1/1 0/0 1/1 0/0 0/1 0/1 
School D 1/0 0/0 1/0 1/0 1/0 1/0 
School E* 1 1 1 1 1 
School F 1 1/1 1/1 1/0 1/0 1/0 
Total 2 8 3 10 7 5 6 
*School E was a case study school only and as such was only visited once 


Following the first year of intervention delivery it was intended to visit all five schools again in order to 
look at the continued implementation of the intervention beyond the required delivery time (after the 
trial’s post-testing). Contact was made with all five schools regarding visits. One school that had been 
visited previously had had a change of headteacher and had stopped delivering the intervention; it 
was unwilling to talk to the research team. Visits were made between November and April 2016 to the 
four remaining schools which included an interview with the headteacher or a member of senior 
leadership (where possible), an interview with the Learning through REAL Projects lead, and a pupil 
focus group and observation if it hadn’t been possible on the previous visit. These interviews explored 
themes brought up by the school on the previous visits as well as questions about sustainability of the 
intervention in the school. 


Control school practice 


Telephone interviews with seven control schools were conducted by one researcher in April 2015 to 
investigate what these schools had been doing through the project. These interviews used a semi- 
structured interview schedule to explore: motivation for signing up to the project, previous experience 
with PBL, current Year 7 teaching including any PBL practice, any plans or training to do PBL the next 
school year, other innovative practice going on in the school, and staff professional development. 


A member of staff from the Innovation Unit was also interviewed regarding which control schools were 
receiving training from the Innovation Unit for the following year (beginning February 2015) and what 
they knew of the PBL practice in the remaining control schools. 


Teacher survey 


An online teacher survey was sent electronically in July 2015 to all intervention teachers involved in 
REAL Projects as provided by the Innovation Unit. This explored teacher views of the intervention, 
how well they were able to implement the intervention, impact on pupils and teaching practice, and 
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views on the support and training provided. Nineteen teachers from six intervention schools 
responded to the survey. This was used to inform understanding of implementation fidelity as well the 
process evaluation findings generally. However, it is worth noting that responses were only received 
from 50% of schools originally allocated to the intervention and this may bias results. 


Developer and intervention delivery team interviews 
Innovation Unit project lead 


A semi-structured interview (with flexibility to explore other topics) was conducted with the two 
Innovation Unit project leads in July 2015 at the end of the intervention delivery by two project 
researchers. This interview discussed: what went well, delivery of the intervention (and variations in 
delivery), perceived impact, assessment of REAL Projects, the accountability framework, and control 
schools’ practices and plans for the future development and delivery of REAL Projects. 


Classroom Coaches 


A focus group with all four Classroom Coaches (employed by the IU to provide a high level of support 
to teachers implementing the intervention) took place in March 2015 to discuss how schools were 
implementing the intervention, the training and support provided, the barriers teachers face, and the 
Classroom Coach role in the project. This was undertaken by two researchers and used a semi- 
structured guide. 


Leadership Coaches 


A focus group was held with all four Leadership Coaches (employed by the IU to support school 
leaders in implementing the intervention) and the Innovation Unit project leads during March 2015. 
This focused on the role of the Leadership Coach and how this supports schools, the 
barriers/difficulties facing school leaders in delivering PBL, how well they perceive schools have 
implemented the intervention, how the intervention had developed during the project, and the 
sustainability of the intervention in schools. 


Case studies 


At the end of the year of intervention delivery, two schools were identified by the Innovation Unit 
project leads where the intervention had been perceived as particularly successful, and those schools 
were invited to contribute to in-depth case studies. Two researchers visited these schools in 
November 2015 to explore practice and to look in more depth at the organisation and delivery of 
REAL Projects. During these visits interviews took place with the headteacher and with the schools’ 
REAL Project lead. Where the visit was the first to a school (as the school was not already part of the 
cross-sectional design) an observation of lesson practice and discussion with teachers delivering the 
intervention also took place. The focus of the case study visits was to look more at the schools’ 
specific REAL Projects practice and to explore in more depth the projects they delivered. The two full 
case studies are included in Appendix E and the findings from these incorporated into the Process 
Evaluation results. 


Fidelity 


Implementation fidelity data was collected through the visits to schools (including observations of 
lessons and interviews with staff), teacher surveys and interviews, and focus groups with those 
supporting schools in implementing REAL Projects. A qualitative description of implementation fidelity 
from across the sample is included in the process evaluation results section. Data on implementation 
was not collected systematically from all schools (five randomly selected schools were visited and 
teachers from six schools responded to the teacher survey), however interviews with the coaches and 
Innovation Unit explored delivery in all schools. 
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The questions and issues explored relating to implementation fidelity were: 


e Did the school meet the minimum implementation criteria? (Data from school visits.) Those 
criteria were that: 
o atleast 20% of the Year 7 timetable should be devoted to the programme; 
0 mixed-ability classes were involved in REAL Projects; 
othe English department should be involved in planning projects; and 
0 planning time should be provided for the REAL Projects team in addition to normal 
planning time. 

e Were schools able to implement the different aspects of a REAL Project? (Data from school 
visits, teacher surveys, and Classroom Coach and Leadership Coach interviews.) Key 
elements were: 

an essential question; 

significant content; 

redrafting and critique; 

a student-created final product; 

an exhibition; and 

oO  anauthentic audience. 
e Did schools receive support as described in the model? Did the support include: 
0 attendance at initial Quest training; 
0 support from a Classroom Coach during the year: and 
0 support from Leadership Coach during the year? 


oOo0o0°0 


Costs 


A specific cost questionnaire was sent to the Innovation Unit which was responded to in detail. In 
addition, more specific cost categories (for example, direct, indirect, in kind, and opportunity) were 
discussed with headteachers and REAL Projects Leads, as a result of which we decided to only report 
the necessary costs to implement and deliver the programme. There was a large degree of variation 
between schools in both how they defined and attributed PBL costs and what they reported spending 
in practice. In some instances schools spent no more—apart from the £10,000 fee to the Innovation 
Unit and additional travel costs—than they would have done had they not done PBL, whereas in 
others there was additional spending on, for example, residential trips for Year 7 pupils. Much of the 
justification for extra expenses did not relate entirely to REAL Projects, but to more general benefits 
such as supporting the transition from primary school. It should be noted that PBL is a whole-class 
intervention and should not necessarily incur any extra costs (for example, to support pupils with 
special educational needs), however each school would need to evaluate whether there were any cost 
implications for its specific circumstances and take the appropriate decisions. 


Most of the training and related costs are ‘up front’, therefore as per the EEF guidance we have 
spread these over a three-year period, however we have not included an inflator as this would 
probably only be 1% or 2% with the main item being professional fees, and other unknown costs such 
as travel would be very variable depending on location and so on. 


An estimation of the time costs for staff attending the initial training is also provided and the 
recommended time for staff planning each week is also provided. However, the actual planning time 
given varied significantly (from no additional time to up to three hours a week) between the schools 
visited for the process evaluation, therefore, it was not possible to give an estimate of the planning 
time necessary to deliver the intervention. 
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Timeline 
The timeline for the evaluation is outlined in Table 3 below. 


Table 3: Timeline for the evaluation 


January 2013 Project start 
April 2013 School recruitment conference for pilot year 
July 2013 Pilot/development training period 


September 2013 Pilot schools begin using Learning through REAL Projects 
November 2013 Telephone interviews, pilot schools 

October 2013-January 2014 | School recruitment, main trial 

January 2014 Randomisation, main trial 

March/June 2014 Observations of main trial training 


May-June 2014 Observation visits and interviews take place, pilot trial 


September 2014 Main trial delivery begins 


April 2015 Control school telephone interviews 
April-June 2015 School interviews and observation visits 1 
June-July 2015 Outcomes testing, main trial 

November 2015-March 2016 | School interviews and observation visits 2 


June 2016 Final report 


July 2014 Piloting outcome measure, pilot trial 
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Impact evaluation 


Participants 


The flow of participants through the evaluation is shown in Figure 1 below. As the recruitment drive 
was directed at all schools across England it is not known how many schools were initially identified 
and approached. Twenty-four schools were initially recruited, however one school withdrew before 
randomisation (it declined to comment on the reason for this), and was replaced by a backup school. 
These twenty-four schools, which did not include any of the pilot schools, were then randomised in 
January 2014. Two intervention group schools withdrew from the project after randomisation, after 
having received some of the initial training but before the start of the new school year. These were 
schools that were under pressure from Ofsted, had undergone leadership changes, and declined to 
respond and engage with the Innovation Unit or the evaluation team. It was not possible to gather any 
data from these schools despite our attempts to contact them. 


Two intervention schools asked to withdraw from the research aspects of the trial wnen approached 
about outcome assessment. One was academised during the year and the Principal and senior 
leadership team left. The new Executive Principal made the decision not to continue with REAL 
Projects as described and was unwilling for pupils in the school to complete any assessments or to be 
evaluated on the way they were implementing the project. The other school disengaged from the 
evaluation before the assessment, although when contacted further they agreed to complete the 
attitudinal survey secondary outcome. They said they were unhappy with the change to the 
intervention only being supported for one year rather than two. 


One intervention school which remained part of the trial asked for opt-in consent from parents to 
undertake the testing. Only a small number of forms were returned to the school (30 out of 125 pupils) 
consequently the evaluation team took the decision that it was not ethical or cost efficient to carry out 
the primary outcome testing in this school. Fifty-five pupils completed the attitudinal survey. 


In total 7 out of 12 intervention schools (1,133 of 2,101 pupils) and all 12 control schools (1,717 of 
1,973 pupils) completed the primary outcome assessment; and 9 out of 12 intervention schools (1,328 
of 2,101 pupils) and all 12 control schools (1,516 of 1,973 pupils) completed the attitudinal survey. 


Issues that arose during the assessment period included schools reluctant to share pupil UPN data 
with the evaluation team (resolved) and schools experiencing restructures and conversion to academy 
status making it challenging to secure their continued involvement in the evaluation. 


Outcome testing was conducted on a single day in each school and pupils who were absent on the 
day of testing were lost. 
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Figure 1: Participant flow through trial 
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As requested by the EEF, a calculation of the actual minimum detectable effect size was conducted 
for different stages of the trial (Table 4). At the analysis stage, using the observed correlation of 0.71, 
the observed ICC of 0.09 (as opposed to the assumed 0.19), the number of individuals included in the 
primary analysis (2,784; 1,119: 1,665), and the average cluster size of 166 gives an effective sample 
size of approximately 2,784 pupils meaning that for this trial we had the ability to detect an effect size 
of 0.36 with 80% power. 


Table 4: Minimum detectable effect size at different stages 


Correlation Minimum 


N [schools/pupils] Average 
(n = intervention; signet og ICC cluster JON) Attrition PN tl 


n = control) eive Ipha effect size 
, post-test (|B) S59) 


24 (12; 12) 
Protocol 3,000 (1,500; 
1,500) 
24 (12; 12) 
Randomisation 4,074 (2,101; 
1,973) 


Analysis (i.e. 19 (7; 12) 
available pre- 2,784 (1,119; 
and post-test) 1,665) 


Pupil characteristics 
School characteristics 


Baseline characteristics of the 24 randomised schools are presented in Table 5; characteristics were 
similar in both allocated groups apart from Ofsted ratings. Ten out of the 12 control schools were rated 
as outstanding or good, whereas 4 of the 12 intervention schools achieved these ratings. 
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Table 5: Baseline school characteristics 


Variable Intervention group lOfol alice) Mel cele e) 


: Hee n/N 
School-level (categorical) n/N (missing) Percentage (missing) Percentage 


200 1200 


Ofsted rating 11/12 (1) 12/12 (0) 
Outstanding 0/11 (0) 2/12 (0) 
Good 4/11 (0) 8/12 (0) 
Requires improvement 5/11 (0) 1/12 (0) 
Inadequate 1/11 (0) 1/12 (0) 
None 1/11 (0) 0/12 (0) 


12.0) 12 0) 


Selection 
Comprehensive 12/12 (0) 10/12 (0) 


Modern 0/12 (0) 2/12 (0) 


Type of school 
Academy converter 1/12 (0) 4/12 (0) 
mainstream 4/12 (0) 4/12 (0) 
Academy sponsor-led 2/12(0) 2/12 (0) 
Community 3/12(0) 2/12 (0) 
Foundation school 2/12 (0) 0/12 (0) 
Free school mainstream 
School-level (continuous) n (missing) Mean (SD) n (missing) Median (min, 
max) 


Number of Y7 pupils 12 (0) 172.5 (43.0) 12 (0) 171.0 (52.8) 


Percentage of pupils eligible 38.5 (25.1, 
for FSM 12 (0) 778) 12 (0) 31.5 (18.3, 56.6) 


Percentage of pupils with 
English as additional 11 (1) 7.3 (1.1, 37.8) 11 (1) 6.6 (1.0, 68.2) 
language 
Percentage of pupils with 
special education needs ee 105 (Gs) 20) ac 
Percentage boys 44 (1) a 12 (0) 52.2 (48.7, 98.7) 


N.B: SD = Standard deviation, min = minimum, max = maximum 


Ertan (12) Rede 


According to their websites, two schools described themselves as already undertaking some project 
based learning: one was allocated to the intervention and the other to the control group. The Ofsted 
ratings were lower in the intervention group compared to the control group, with ten intervention 
schools rated as ‘requiring improvement’ or ‘inadequate’ and only two control schools in the same 
categories. Most of the control schools were rated as ‘good’ or ‘outstanding’. The mean school size 
was around 170 pupils in both groups. The median percentage of pupils eligible for FSM was slightly 
higher in the intervention group (38.5) compared to the control group (31.5); this is higher than the 
national average of pupils known to be eligible for and claiming free school meals in state-funded 
secondary schools which was 13.9% in January 2015 (Department for Education, 2015). The median 
percentage of pupils with English as an additional language was slightly higher in the intervention 
group (7.3) compared to the control (6.6). The percentage of pupils with SEN was approximately 11% 
in both the intervention and control groups. The median percentage of boys was similar in both groups 
at just over 50%. 
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Pupil characteristics 


Characteristics of 3,483 pupils from participating secondary schools are presented in Table 6. As this 
was a Cluster randomised trial, randomisation aimed to balance the trial arms with regard to cluster- 
level characteristics rather than individual-level characteristics. The mean KS2 scores were similar 
between intervention and control groups across all components in both the randomised and ‘as 
analysed’ samples. There was a slightly higher percentage of males in both the randomised and ‘as 
analysed’ samples, and a slightly lower percentage of FSM pupils in the control group compared to 
the intervention group in the randomised sample and the reverse for the ‘as analysed’ sample. There 
were disproportionate amounts of missing data for the KS2 baseline measures between the 
intervention and control groups, 35% and 7% respectively. This was in part related to the withdrawal 
of two intervention schools and 317 associated pupils prior to commencement of the intervention. 
Given schools were randomised before pupils were attending the secondary school it was not 
possible to identify pupils baseline characteristics prior to the withdrawal. 
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Table 6: Baseline pupil characteristics 


Intervention Control 
n/N (missing) Percentage 


Percentage 


43.0% 688/1,847 (126) 37.3% 


776/1,582 (519) 808/1,891 (82) 
806/1,582 (519) 1,083/1,891 (82) 


Pupil-level (categorical) n/N (missing) 
As randomised 


Eligible for FSM 


Gender 
Female 
Male 

As analysed 


Eligible for FSM 


666/1,549 (552) 


34.4% 595/1,665 (308) 35.7% 


49.4% 736/1,665 (308) 44.2% 
50.6% 929/1,665 (308) 55.8% 


Mean (SD) n (missing) Mean (SD) 


19682 


n/N (missing) Percentage n/N(missing) Percentage Effect Size 


666/1,549 (552) 43.0% 688/1,847 (126) 37.3% -0.04 


776/1,582 (519) 49.1% 808/1,891 (82) 
806/1,582 (519) 51.0% 1083/1,891 (82) 


385/1,119 (982) 


553/1,119 (982) 
566/1,119 (982) 


Pupil-level (continuous) | n (missing) 


As randomised 


As analysed 


KS2 Average point score 


Pupil-level (categorical) 
Eligible for FSM 


Gender 
Female 
Male 


Pupil-level (continuous) 


KS2 Reading (pre score) 
KS2 Mathematics 
KS2 Average point score 


Outcomes and analysis 


1,548 (553) 
1,548 (553) 
1,548 (553) 


1,119 (982) 
1,119 (982) 


Mean (SD) n (missing) Mean (SD) 


28.1 (4.6) | 1,836 (137) | 28.1 (4.4) 0.09 
28.4 (5.0) | 1,836(137) | 28.3 (4.9) 0.01 
28.1 (4.3) | 1,836(137) | 28.1 (4.2) 0.0006 


n (missing) 
1,548 (553) 
1,548 (553) 
1,548 (553) 


Summary of raw outcomes 


Raw unadjusted means of PiE scores are presented in Table 7 by intervention and control groups. 
Mean PiE scores were similar in both allocated groups at 24.2 (SD 11.2) in the control group and 24.5 
(SD 10.8) in the intervention group. Proportions of individuals with missing data on PiE12 scores 
and/or covariates included in the analysis were 16% and 47% in the control and intervention groups 
respectively. The 47% attrition rate in the intervention group is unusually high and could be seen to 
indicate a biased drop out due to the intervention. However, from the five schools for which data was 
missing, two schools withdrew before starting the intervention due to a change in leadership and 
school priorities which was not related to the intervention, while an additional school did not provide 
follow up data due to a lack of parental consent to conduct the assessment, again unrelated to the 
intervention. The final two schools did implement PBL during the year. One of these schools refused 
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to sit the tests because they did not think an effect would be evident after one year. In the other 
school, the whole leadership team was replaced by a new one under an academy sponsor. The 
secondary outcomes of attitudinal survey total scores and attendance were also similar in the two 
groups. 


Table 7: Raw summary statistics for outcome scores 


Raw means 
Intervention group Ofoy nigel me | celyy e) 


n 


Outcome (missing) 


Mean (SD) n (missing) Mean (SD) 


Progress in 
English 12 1,133 (968) 24.5 (10.8) 1,717 (256) 24.2 (11.2) 
Attitudinal survey 
‘otal scores 1,124 (977) 164.3 (21.5) 1,262 (711) 163.7 (22.7) 
Attendance 1,573 (528) 0.96 (0.05) 1,899 (74) 0.96 (0.05) 


Intra-cluster correlation coefficients (ICCs) and correlation 


An intra-cluster correlation coefficient (ICC) was estimated in relation to the primary outcome of PiE 
scores and found to be 0.06 (95% Cl: 0.02 to 0.09) for 2,850 pupils. The correlation between the 
primary outcome of PiE and KS2 reading fine grade scores (pre-test) based on 2,784 pupils was 
estimated to be 0.71 which was as anticipated in the sample size calculation. 


Regression model results 


Table 8 and Table 9 show summary results for primary and secondary regression analyses. No 
significant differences were found between the two randomised groups in any of the primary or 
secondary analyses. Results are discussed in more detail in the following sections. 


Education Endowment Foundation 32 


Project Based Learning 


Table 8: Primary analysis 


vie eee , Effect size p- 
(intervention; 


1) 
control) (95% Cl) WEVLUT=) 


Progress in 2,784 (1,119; -0.13 (-0.31, 


Table 9: Secondary analyses 


Score difference (95% 
(o)) 


Outcome 


Score difference (95% UM Effect size os 


(intervention; a 
1e4)) control) (95% Cl) value 


Outcome 


Repetition of primary 


analysis adjusting for -1.05 (-2.36, 0.26) er eN IA: een ee 


1,665) 0.04) oe 


gender 


Attitudinal survey total i 2,338 (1,108; 0.01 (-0.15, 
Attendance (transformed) -0.012 (-0.04, 0.02) a aie ~ 


Primary analysis 


The primary analysis was adjusted for baseline KS2 reading fine grade score, FSM status, number of 
Year 7 pupils, whether a free school or not, and whether the school advertised using a form of Project 
Based Learning prior to trial. After exclusions for missing data relating to any of these variables or the 
response, analysis was conducted on 2,784 pupils (1,119 in the intervention and 1,665 in the control 
group). There was little or no evidence of a difference in PiE scores, with a non-significant decrease of 
1.00 point for those in the intervention group compared with those in the control group (p = 0.15, 95% 
Cl: -2.38 to 0.37). This relates to an effect size of -0.13 (95% Cl: -0.31 to 0.05). 


Secondary analyses 
Repetition of the primary analysis with an additional adjustment for gender 


This analysis was conducted on the same 2,784 pupils as those for the primary analysis. There was 
little or no evidence of a difference in PiE scores, with a non-significant decrease of 1.05 points for 
those in the intervention group compared with those in the control group (p = 0.12, 95% Cl: -2.36 to 
0.26). This relates to an effect size of -0.14 (95% Cl: -0.32 to 0.04). 


Attitudinal survey total scores 


This analysis was conducted on 2,338 pupils (1,108 in the intervention group and 1,230 in the control 
group). There was little or no evidence of a difference in the attitudinal total scores, with a non- 
significant increase of 0.28 points for those in the intervention group compared to the control group (p 
= 0.87, 95% Cl: -3.12 to 3.68). This relates to an effect size of 0.01 (95% Cl: -0.15 to 0.18). 


Attendance data 


This analysis was conducted on 3,381 pupils (1,546 in the intervention group and 1,835 in the 
control). This data was requested and supplied from the NPD. After investigation of model 
assumptions it was identified that the normality assumption was in question and the residuals were 
highly skewed. Given the nature of the outcome variable and the type of skewness observed, the data 
was transformed using an arcsine transformation. Model assumptions were repeated after including 
the transformed variable in the analysis. The resulting distribution of the residuals was closer to 
normality so the transformed variable was used in the analysis in order that the confidence intervals 
and p-values could be interpreted with more confidence. There was little or no evidence of a 


Education Endowment Foundation 33 


Project Based Learning 


difference in the attendance, with a non-significant decrease of 0.012 points in the transformed 
variable for those in the intervention group compared to the control group (p = 0.46, 95% Cl: -0.04 to 
0.02). This relates to an effect size of -0.08 (95% CI: -0.29 to 0.13). 


Subgroup analyses 


Three pre-specified subgroup analyses (FSM status, gender, and KS2 reading fine grade scores) 
included an interaction term between each baseline variable of interest and group allocation for the 
primary outcome only (Table 10). There was no evidence of a statistically significant interaction 
between allocated group and gender (p = 0.49; 95% CI: -1.54 to 0.74) in terms of PiE12 scores, nor 
allocated group and prior attainment (p = 0.39; 95% Cl: -0.08 to 0.20), suggesting the intervention did 
not have a differential effect dependent on these two variables. 


There was a statistically significant interaction in PiE between allocated group and FSM status (p = 
0.02; 95% Cl: -2.77 to -0.30) suggesting the intervention had differential effects dependent on FSM 
status. This relates to an effect size of -0.38 (95% Cl: -0.38 to -0.02). As requested by the funder, the 
primary analysis was repeated using data only from pupils eligible for FSM as a subgroup analysis. 
There was evidence of a difference between allocated groups in PiE between those eligible for FSM 
randomised to the intervention and control groups with a significant decrease of 1.85 for pupils in the 
intervention group when compared with those in the control group (p = 0.009; 95% CI: -3.25 to -0.46). 
This relates to a statistically significant effect size of -0.24 (-0.41 to -0.06). 


Table 10: Subgroup analyses 


Te] ole cel le) Effect size (95% Cl) p-value 


FSM status interaction -0.38 (-0.38 to -0.02) 
FSM status subgroup -0.24 (-0.41 to -0.06) 


KS2 reading fine grade 0.04 (-0.17 to 0.19) 
scores 


An exploratory subgroup analysis for FSM status for the secondary outcome attitudinal total scores 
was also undertaken. There was no evidence of a statistically significant interaction between allocated 
group and FSM status in terms of attitudinal total scores (p = 0.69; 95% CI: -2.92 to 4.44). 


Missing data analysis 


Given the proportion of missing data, both at baseline and at post-test PiE12 scores (n = 1,290) we 
conducted a sensitivity analysis using multiple imputation by chained equations to include all schools 
and pupils randomised (Royston and White, 2011; Azur, 2011). The results from the sensitivity 
analysis were similar to those from the primary analysis ( 
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Table 11). There was little or no evidence of a difference in PiE scores, with a non-significant 
decrease of 0.88 point for those in the intervention group compared with those in the control group (p 
= 0.10, 95% Cl: -1.95 to 0.18). This relates to an effect size of -0.11 (-0.24 to 0.02). 
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Table 11: Missing data analysis for the primary outcome 


ay tees ‘ Effect size p- 
(intervention; 


tt) 
control) (95% Cl) NEVLUT=) 


Primary 1.00 (-2.38, 0.37) es oe ea ca ' 
imputed -0.88 (-1.95, 0.18) ‘oa a Co. io 


Cost 


Progress in Score difference (95% 
English 12 (o3)) 


For the calculation we have assumed a six form entry school, and that PBL is delivered to all of the 
pupils in Year 7 over the three-year period and continues into the following year. 


Participating Pupils 
Year 1 Year 2 Year 3 Total 
Y7 180 180 180 540 
Y8 180 180 360 
900 


The first year cost per school is £22,950 (reduced to £10,000 for the trial schools). The Innovation Unit 
has advised that following years would be £10,000. Travel costs by the Classroom and Leadership 
Coaches would be in addition. These were capped at £3,000 for the trial and we have also used this 
figure for future years. This element is highly variable depending on the location of the school and 
number and location of coaches. 


Year1l | Year2 | Year3 | School Total | Cost per pupil 
Actual 25,950 13,000 | 13,000 51,950 £58 
Subsidised | 13,000 | 13,000 13,000 39,000 £43 


The figures do not include VAT which the Innovation Unit charges, and schools would have to provide 
cover for teachers attending the QUEST training. Assuming 3 teachers attending, this would require 
nine days of cover, as well as travel and accommodation costs. The intervention also specifies a 
requirement for additional planning time for the team delivering REAL Projects which is recommended 
to be between one and three hours per week. In practice this planning time varied hugely between no 
specific time and up to two hours per week. 


In summary, we have therefore assumed the cost per pupil to be £58 (+VAT), with additional school 
costs associated with nine days of cover. The additional time in schools for planning and meetings 
related to PBL have not been included due to the variation between schools and it being reasonable 
to assume that additional time would be needed for any new initiative or ongoing development. 
Whether such additional costs are considered part of the PBL budget would, of course, be a decision 
that each school would have to make in light of its own circumstances. 
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Process evaluation 


Fidelity 
Intervention Group 


Implementation fidelity was assessed for the ten intervention schools that delivered the intervention 
(the two other intervention schools withdrew from the project after receiving some initial training but 
did not deliver any of the intervention). Fidelity to the REAL Projects intervention was reasonably high 
considering the necessary variation and adaptations to the different school contexts. Schools 
delivered between four and six projects to Year 7 pupils during the year with all projects covering 
more than one subject area. 


In terms of minimum organisational elements, all of these schools started the year delivering at least 
20% of their Year 7 curriculum using REAL Projects as required. One school, following a change of 
leadership in February, reduced this to below 20% (changing staffing and planning time for projects at 
the same time). All schools, apart from one, began the year with mixed ability REAL Projects classes. 
That one school, however, which started out using sets subsequently switched to mixed ability REAL 
Projects classes, whereas two other schools reverted to setting after their first project. The English 
department was involved in planning projects in all schools, but the extent of the involvement varied 
from advising on assessment for already-planned projects without involvement in teaching, to being 
involved in the full planning process and teaching REAL Projects. Not all schools were able to give the 
required amount of additional planning time to REAL Projects teachers: two intervention schools gave 
no additional regular planning time, and half of the schools (who responded to the teacher survey) 
only provided up to 30 minutes a week. All schools sent teachers to attend the initial Quest training, 
however some teachers that attended this training had left the school by the start of the delivery year. 
New teachers were trained by the Classroom Coach instead (from the support time allocated to the 
school). Support from the Leadership Coaches and Classroom Coaches was provided as described in 
the model. 


Delivery of the REAL Projects generally included the key elements specified, although the Classroom 
Coaches and our school visits showed that the quality of these elements varied between schools. All 
schools were using an essential question to drive the content of a project and were strong on using 
critique and redrafting with particular focus on peer critique. All schools delivered projects which 
included rigorous content, based either on the curriculum for that subject or on the project topic, 
however the level of rigour varied from project to project; Classroom Coaches indicated that 
sometimes the rigorous content was taught as an add-on to the project rather than through the 
project. All schools had planned to have a student-created final product, however on some projects in 
a few schools this was not possible (for example, publishing a finished book) due to budget 
constraints or other organisational factors. Schools did generally deliver an exhibition at the end of 
each project but in most cases this was for parents or other pupils in the school rather than a public 
exhibition with a truly ‘authentic audience’. There were, however, a few strong examples—such as art 
displayed in a local art gallery and a local artist invited to provide feedback—but these were the 
minority. Schools reported this to be a difficult aspect to get right due to the need to make connections 
outside of school and the time taken to set it up. 


Control Group Activity 


A number of schools in the control group were delivering some aspects of project based learning 
during the year of the trial. Four schools in the control group were delivering more than 20% of their 
Year 7 curriculum through PBL, and two of these four schools paid to have external professional 
development and support from other organisations: one paid for staff to visit and learn from High Tech 
High (a PBL-based school in the U.S.) and for support from EOS Education (a PBL teaching alliance). 
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Five control schools signed up to deliver REAL Projects the following year and therefore worked with 
the Innovation Unit to plan for this and to receive training from January/February 2015 as agreed, 
however two of these schools also implemented one or two small trial projects towards the end of the 
year with Year 7 pupils. Six control schools did not deliver any aspect of PBL during the trial year 
(except the training for the following year). 


It is worth highlighting that during an interview with a control school the senior leader explained that 
the school had signed up to the RCT because it was interested in delivering the intervention. It had 
therefore been willing to pay the £10,000 required as part of the trial because it was felt that it was the 
best way to support pupils in the school. Although the school had signed an MoU for the project 
committing it not to deliver PBL, it felt the needs of its pupils came first and would be best supported 
by delivering PBL, despite the MoU agreement. 


Interviews with Innovation Unit staff revealed that it was unlikely that the control schools were 
delivering PBL in the way that REAL Projects prescribed (with the exception of one school with 
significant paid support) and even schools paying for some support would not be receiving the extent 
of support provided by the Innovation Unit. 


However, this control group activity limits the conclusions we can make from the trial given that a third 
of the control group were delivering some aspects of PBL for 20% of the Year 7 curriculum while 
another two control schools delivered some aspects of project based learning towards the end of the 
year. 


Implementation 


The following sections look at issues of implementation, barriers to delivery, outcomes, and formative 
findings across five schools. We consider insights gained from two case-studies carried out at two 
schools where it was agreed—by both the Innovation Unit, as developer and delivery partner, and the 
evaluation team—that the implementation of REAL projects had been effective. Both schools also 
agreed that their implementation had been effective. Importantly, while there are quite substantial 
similarities between the schools, the implementations were nonetheless different in a number of 
respects—supporting the notion that there is no simple ‘one size fits all’ version or implementation of 
PBL. Some background information on these two case studies is presented next. 


Case studies: background to the schools 


Both schools are larger than average, well established, and draw the majority of their pupils from 
nearby communities on the outskirts of large conurbations. There were relatively small differences in 
their last Ofsted ratings—2 (‘good’) for School E, and 3 (‘requires improvement’) for School F, with 
one having gone up and the other down—but nonetheless both were judged to be reasonably 
effective with good leadership. In both instances, English teaching at KS3 was identified as an area 
that could be improved—the primary outcome for the trial. Attitudes to learning and support from 
parents were judged to be good, as was attendance and relationships between pupils and teachers. It 
can therefore be reasonably assumed that from an Ofsted perspective both schools were in a good 
position to implement and manage the necessary organisational changes required for PBL. 


School E 


The starting point for school E was an ongoing internal review of their curriculum and wider school 
development, and an identified need to improve KS3 teaching. Some elements of ‘project work’ and 
student-led or independent learning were already being undertaken and the school as a whole was 
keen to encourage similar initiatives. In such circumstances it was not surprising that the offer to be 
included in the trial, in particular the intervention arm, was enthusiastically accepted. The headteacher 
emphasised that the school was looking for something like PBL to provide more structure and rigour 
to their plans and aspirations for KS3. 
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School F 


School F was motivated to do REAL Projects in order to holistically improve educational outcomes for 
pupils leaving secondary school, both in terms of exam outcomes but also promoting skills required 
for employment and further or higher education (including in particular STEM subjects) as well as local 
needs. It felt the architecture for improvement was largely already in place, but that the curriculum, 
both nationally and how it was being interpreted locally, was not able to support the outcomes it 
desired. The school had a long term interest in initiatives such as the CBI ‘21st Century Skills’ and the 
more recent RSA ‘Opening Minds’ competencies framework, which mesh well with the underlying 
philosophy and principles of PBL. As the head and PBL lead emphasised, pupils come from primary 
schools with a passion to learn and it is essential they put in place a way for that passion to continue 
and thrive in secondary school. 


The necessary conditions for the success of the intervention 
Belief in PBL principles 


A major enabler for the success of the intervention was the schools’ desire to take project based 
learning on board and a deep belief in the principles underlying this type of learning. Schools believed 
that pupils needed to be actively involved in their learning and develop a passion for learning—skills 
essential for developing learning and competences throughout life that would adequately meet 21st 
century workplace requirements. All five schools held a deep-seated belief that education needed to 
change to incorporate higher quality learning able to enhance pupils’ ability to be critical thinkers and 
allow them to be creative in an environment where learning is relevant and has a real purpose. The 
quotations that follow illustrate the schools’ belief in the project-based learning principles which drove 
its desire to sign up for the initiative: 


‘...1 was crying out for something that would formalise the way that we were doing 
competence-based learning and give our teachers the pedagogical framework for enabling 
what we wanted. The core things that | wanted were basically a toolkit, and a lexicon around 
‘learning to learn’ (School F, headteacher). 


‘they need to be lifelong learners because what they need to know and understand now, 
and what they'll need to be able to know and understand and the skills they'll need to apply in 
2, 3 years’ time, will be completely different’ (School B, PBL lead). 


‘We wanted to be outstanding for attitudes to learning here, [and] for behaviour. One of the 
reasons that we thought REAL would be good was pupils’ involvement in their learning... so 
they would be active learners rather than just kids sitting quietly in a classroom’ (School A, 
headteacher). 


‘l really do believe in the real learning opportunities that this presents for the pupils—for all the 
pupils to engage—because | think engagement is critical... | also think that [in addition to] the 
projects giving the relevance to their learning, it actually helps pupils to engage; and if you’ve 
engaged pupils and got them motivated then you're halfway there to getting them to achieve, 
and so | am absolutely an advocate of REAL Projects’ (School C, headteacher). 


Strong team, collaboration, and time given for PBL 


All schools agreed that a strong team of staff would be essential for the success of project based 
learning and this was an area that would be within the control of the school. A good leader supported 
by senior management, and an open-minded team of teachers who believe in the principles of project 
based learning and are determined to see it working, were seen to be key factors in facilitating the 
implementation of the intervention. 
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‘| couldn't have been luckier with the team as the person leading it... [the team] have been 
incredibly enthusiastic, and because they’ve got different specialism, it’s made it a lot easier, 
especially having the Art specialist, because a lot of the projects have creative outcomes’ 
(School C, PBL lead). 


Keeping the lines of communication open and exhibiting a real team spirit can create a ‘cohesive 
team’ that shares the same teaching and learning aims as well as sharing expertise and subject 
knowledge: 


‘It’s a very supported curriculum in terms of project handbook, team training... a lot of 
emphasis on us working together and reflecting, reviewing and that’ (School F, follow-up 
meeting, headteacher). 


Having planning time was one of the challenges raised by staff, particularly during the first year of the 
project. In most cases, schools had found ways to address this in the second year of the project and 
had managed to protect some time to be spent solely on PBL planning. Planning time was seen as 
important for the success of REAL Projects in schools, particularly as this gave a chance for teams 
from across different departments to work together and for organising resources, such as setting 
dates for key events and booking rooms. 


‘Towards the end of last year we were given four days off timetable where we had a core 
team of five of us that went off-site. We sat there and planned the projects and that was 
brilliant because then it meant you could start this year with every single exhibition date on the 
calendar’ (School B, follow-up visit, PBL lead). 


Last but not least, it was felt that an important part of the PBL process was for schools to reflect on the 
success of each project delivered and to look at the modifications and alterations that they could make 
to improve it for the future: 


‘It has been a journey we’ve been on since the beginning. The second project didn’t work as 
well as the first project and then we learnt from that, and there was much more of the learning 
diaries and the learning logs and the progress logs which made a real difference’ (School A 
headteacher). 


Importance of the PBL process and product 


The essential elements of drafting, redrafting, and critiquing as part of the process to a final product 
and exhibition with audience were seen to be very important. The process was described as a very 
creative one where pupils were involved in an iterative and dynamic process of exploring, thinking, 
reflecting, reconsidering actions, and making decisions before the final product could be constructed 
and presented in public. 


It was also important for the reflection and critique to be captured and assessed. This was an aspect 
of pupils’ learning that lent itself to making clear connections with literacy work. This writing process 
had a two-fold purpose. First, it served as a tangible document that pupils could use to reflect on, draft 
and redraft throughout the project and, second, it was a useful assessment tool that could be used to 
demonstrate the progress that the pupils were making in writing. 


‘...-having that critique and constantly coming back and saying, is this good enough? How 
could we improve it? What else do we need to do to add to it? That’s what keeps driving it 
forward’ (School A headteacher). 


Within the context of exhibiting pupils’ work, it was felt that the expectation that the final product 
should be of high quality needed to be cultivated and taught. Some schools recognised that they 
needed to provide support to help pupils develop these high expectations: 
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‘I’ve still got some work to do with some of my teachers in valuing excellence in terms of 
outcome, that actually that is a really important thing to model: it’s not just about learning, it’s 
about creating something with excellence and that breeds pride in pupils which they dont 
necessarily have’ (School D headteacher). 


For pupils to fully benefit from the REAL Projects model, it was felt that the final outcome needed to 
be valued and judged by an ‘authentic audience’. Schools generally stated that involvement of parents 
and members of the community can help create this audience which is essential for the successful 
completion of the project. Schools recognised this and put extra measures in place to engage more 
parents in these exhibition events in the second year of the project. 


Training and support 


The good training and constant support provided by the Innovation Unit was very much appreciated 
and recognised by all schools. The initial training was very informative and offered a number of 
inspiring project ideas. The coach’s support and guidance were essential for the projects to take off, 
and coaches provided an important source of advice at times when difficulties were encountered 
along the way. Further training was being planned for members of staff who did not get the chance to 
attend the initial training. This was usually being done in schools either by trained and experienced 
staff or by the Classroom Coach. 


Training and Innovation Unit support in School E 


The school was impressed with, and highly valued, the support from the Innovation Unit and the 
Leadership and Classroom Coaches, although in different ways. The Classroom Coaches were found 
to be very approachable and knowledgeable, and importantly were happy to work alongside the 
teachers. The Leadership Coach needed to adopt a role ‘sympathetic’ to the needs of the 
headteacher and leadership team, and this school felt that this had been achieved. 


In addition to two of the staff attending the QUEST training in London, the Classroom Coach 
replicated the programme in the school for most of the REAL Projects teaching team. This took place 
over a two-day period and was found to be highly productive and beneficial. Importantly, this helped 
bring the staff together for PBL in a way that having just one or two teachers reporting back to the rest 
of the staff would not have done. 


It was felt that the Classroom Coach had unique skills and experience that were unlikely to be 
available currently in U.K. based trainers. However, the Leadership Coach skills were viewed as more 
generic, and could therefore be ‘bought in’ locally or regionally. It was also suggested that some of the 
school’s PBL staff would be able, in time, to fulfil a coaching role by sharing their PBL skills and 
expertise with other schools. 


Training and Innovation Unit support in School F 


Two teachers from School F and one member of the senior leadership team attended the QUEST 
training in London. This was found to be very helpful and in particular helped the school to identify 
ways in which their educational aspirations could be realised as well as training them to effectively 
implement REAL Projects. The cost of attending this, both time and expenses, were described by the 
headteacher as significant ‘front loaded’ costs which it was not possible to continue funding beyond 
the first year of the project (despite continuing with PBL). Therefore, the following year the QUEST 
training was done in the school by the Classroom Coach and the PBL lead. This approach helped the 
school to tailor the project and training more to local needs although while still retaining the essential 
elements of PBL. 


The input of both the Leadership and Classroom Coach was very much valued by the school and PBL 
team. Both worked mainly with the PBL lead as the role of the Leadership Coach, which was normally 
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to support the head to establish the right structure and conditions for PBL to work, was not really 
necessary given what the school was already doing. Moving forwards, the school would be happy to 
purchase support as required, but not necessarily the whole package. 


At quite an early stage the school was encouraged by the Innovation Unit to help other schools adopt 
the programme given their positive experience. However, they took a cautious approach to this, 
concerned that the particular expertise of the Classroom Coaches would not be that easy to replicate, 
and also wary of not paying enough attention to their own needs. The PBL lead has since gone on to 
do some support and consultancy work with another school, but is still able to support the project in 
her own school, and has benefited professionally from this. 


Barriers to delivery 


Assessment 


Uncertainty around assessment and how it fits with project based learning was evident in all five 
schools. One of the major challenges for school staff was demonstrating that pupils could make 
measurable progress within an educational culture of testing and accountability to a level that would 
satisfy governmental expectations. As a result, assessment created a tension for the schools which 
sometimes led to having to compromise innovative ideas about assessment in order to satisfy 
perceived Ofsted requirements. This was a major concern that was voiced in all schools. 


‘We need to make sure there’s data, we need to show progress, we need to show that the 
kids are improving their literacy and we are meeting the Ofsted criteria because otherwise we 
havent got a leg to stand on and we will be slashed very quickly’ (School B, PBL lead). 


‘We just want to get past Ofsted. That’s a shame because it stops things moving forward, but 
we just need to do that first’ (School C headteacher). 


The assessment of pupils was perceived as needing to be aligned to the nature of project based 
learning but also needing to conform to traditional notions of testing and measuring progress. One 
school tried to achieve this by taking a ‘holistic approach’ to assessment where projects would be 
marked as a whole without compartmentalising different subject areas. 


A desire to establish consistent and rigorous assessment systems was common, and different ideas 
emerged across schools for doing this, however more time and development was still needed for 
these to be purposefully applied. A common view was that appropriate assessment would lead 
learners to be more independent, more critical of their own and others’ work, and guide learners to 
make improvements in their work. There was also a perceived tension between the evident positive 
outcomes regarding pupils’ communication and presentation skills and the schools’ perceived difficulty 
in quantifying this systematic progress in learning—to demonstrate a ‘tangible learning experience’ 
(School B, Deputy Head). Assessment of project based learning was, therefore, an area in which all 
schools were trying hard to improve as the following comments illustrates: 


‘But there is no clear assessment from the outset and we are developing it... wishy-washy is 
the wrong word but it’s... not solid, it’s not shown to kids as I'd like to, and parents haven't got 
a clue. So assessment does need looking at. We have started’ (School B teacher). 


Many schools were using essay-writing to provide a tangible assessed product for every project. 
Schools were trying to establish assessment criteria for this written project-related work within the 
context of general assessment practices in the school. Even though-essay writing seemed to be one 
way forward in assessing PBL, concerns were raised about the challenges in marking essays and the 
importance of doing this ‘properly’. Furthermore, essay-writing as the only option in demonstrating 
rigour in assessing PBL was questioned. These points are illustrated below: 


Education Endowment Foundation 42 


Project Based Learning 


‘There does seem to be a bit of a link with rigour and writing essays [but] that’s not the only 
way to demonstrate something is rigorous. There’s obviously lots of different opinions being 
thrown into the pot and that is one gripe of mine at the moment—! don’t think rigorous 
education needs to just be essay-writing. | don’t think they need to do it in every single project’ 
(School C, PBL lead). 


Because of these challenges around assessment, schools were keen to attend relevant assessment 
forums in London where ideas about good practice and success stories would be shared. 
Headteachers expressed the desire to allow members of staff to attend these forums to bring back 
innovative assessment ideas which could be considered for implementation in their school. 


Practical limitations 


Limited ICT resources was another hurdle mentioned in the interviews that made the initial stages of 
PBL implementation quite difficult. Problems were encountered when there were not enough ICT 
classrooms available or when the iPads or iPhones that some pupils had did not have the necessary 
resources installed for purposeful PBL work to take place. The limited resources would limit the quality 
of the work that the pupils could produce. 


‘For us in our first half term where it was the first project that came from the Innovation Unit 
we had to do a book. We just did not have the IT resources—it was an absolute nightmare. It 
was a trial. The way it was implemented did not work for us at all, the way we’d done it—not 
the way they’d done it. Logistically—because we didn’t have the ICT facilities—how were we 
supported to produce a professional-looking book?’ (School D teachers). 


Other challenges, such as having suitable space for project-based work and for the public exhibition, 
time pressures to organise meetings, having enough time as a team to plan appropriate work, 
preparation time, timetabling, and financial constraints were all mentioned as organisational issues 
that limited the extent that PBL could be optimally delivered. Moreover, specific issues within schools 
to do with time pressures, and the expectation that teachers should successfully fulfil a number of 
different roles, meant that, occasionally, some suggestions given by the trainers about how to run a 
project in an efficient way could not always be realistically implemented. 


Staff perceptions in the rest of the school 


In a few schools, staff involved in teaching and leading REAL Projects felt that there was resistance to 
PBL from other teachers in the school and a lack of understanding of the principles of PBL. This 
meant that it was difficult to get the principles embedded in other areas of the school and sometimes 
led to the REAL Projects team feeling isolated, especially when the majority of their teaching time was 
in REAL Projects. 


‘I think one of the mistakes we’ve made is not managing to make it as clear to other staff as it 
could be what it’s about and spreading it throughout, so | think it would be fair to say there’s a 
lack of understanding from staff not involved directly in PBL. We’ve managed to get some 
involvement from the Directors of Maths and Science, so there’s elements of PBL coming 
through other areas of the curriculum, but | would say there’s not a whole-academy 
understanding really of what PBL is and that’s something that we need to tackle in the coming 
year’ (School B, PBL lead). 


‘If we were to have had this year again | think we would have done more CPD work with the 
whole staff, so that the whole staff's understanding was better... so | think we’re probably 
much more knowledgeable now to do it with the whole staff next year’ (School A head- 
teacher). 


Education Endowment Foundation 43 


Project Based Learning 


Schools were therefore aiming to tackle this in the second year of delivery and were putting more 
emphasis on a whole-school approach where more members of staff, even those not directly involved 
in PBL, would become familiarised with aspects of it. They hoped that PBL principles would permeate 
the whole school ethos. 


Organisation and challenges in School E 


In School E, both the headteacher and the PBL lead, as well as several of the teachers we spoke to, 
reported that one of the biggest challenges of doing the PBL project was the very tight timescale. In 
practice this amounted to signing up in February with training taking place in May or June and starting 
to deliver REAL Projects the following September. This was necessary for the timing of the trial, but 
there was a consensus that at least a term or two is needed to plan for the necessary organisational 
changes, and that ideally a year is needed to effectively prepare to implement REAL Projects. It 
follows that quite a few schools would be unable to undertake the necessary preparatory work in this 
short time-period; that this school was able to is a reflection on the school, and should be considered 
when generalising. The school felt in a secure position internally and externally to address the 
challenges of a major new initiative such as PBL. 


A major perceived challenge identified early in the planning phase was getting the necessary ‘buy-in’ 
and support from three departments not used to working together. They found that the departments 
were able to work well together, but they emphasised that it was important to ensure that all three 
were equally valued and that time taken for REAL Projects from subject teaching was proportionate. 
Once the project was underway, most concerns centred on logistical issues. While it was felt that 
better forward planning would have helped, in practice it was felt that any major organisational change 
would have presented similar challenges. It was also emphasised that many staff needed additional 
support when working outside of their ‘comfort zone’-—such as having to work in a cross-curricular 
way, cope with new subjects, or use ICT more intensively. Noting that Year 7 pupils are generally 
enthusiastic about their new secondary school experience, it was emphasised that the ‘student voice’ 
survey showed particular enthusiasm for project-working (comparison was made to the previous year 
as well as other non-PBL subjects). 


An important commitment was made at an early stage of planning that the curriculum needed to drive 
the timetable, even when this might have knock-on effects on other subjects. In practice, the REAL 
Projects curriculum was not found to constrain wider school working significantly. The pupils did PBL 
twice a week for 150 minutes at a time (triple 50-minute lessons). One session was humanities- 
focused and the other business and ICT. This was found to be a good and effective arrangement 
which fitted well with the wider school organisation and will continue for future years. 


The staff we spoke to emphasised the value and importance of headteacher, senior staff, and 
governor buy-in to the pedagogy. Again, this largely reflects the nature and values of the school and 
importantly was identified by everyone we spoke to at the school as an essential prerequisite for 
effective PBL. There was quite considerable emphasis on the need for planning and meeting time, in 
particularly as there were up to 12 teachers delivering PBL. The school also placed considerable 
importance on the role of teaching assistants (TAs) to support the pupils generally as well as 
individually for those with specific needs. Indeed, we saw one lesson where the pupils had reading 
ages ranging from 6 to 15 years, as well as EAL pupils, and in such circumstances TAs played a very 
important role. 


Organisation and challenges in School F 


In School F, in addition to having much of the organisational structure in place for PBL, at the time 
when the Innovation Unit approached the school to join the trial the PBL lead had recently completed 
a Masters degree focusing on how STEM could be effectively taught at the school through PBL. This, 
combined with support from the governors, made it relatively straightforward (compared to other 
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schools) to implement PBL. However, they were aware that it did not really fit clearly with government 
policy in practice, or with ‘playing safe’ with Ofsted. They suggested that the uncertainty relating to 
PBL was a perception that there was a lack of empirical evidence supporting it as an educational 
approach. However, this is also the case with a lot of educational approaches, for example, those 
concerned to address the issue of white working-class boys performing below their peers. 


The school benefited from good community links and at that time teachers were organised on a 
faculty basis, rather than along strict subject lines. Both of these factors helped with the organisation 
and implementation. Although PBL was a high-profile project, the school sought to play this down 
internally so as to reduce the chance of some element of resentment. The PBL teachers were 
organised on a small team basis, with most of their teaching being PBL. Overall the school sought to 
‘grow’ the teachers in to their role, and had great success with new or NQT teachers in the PBL team 
who were able to get support from the lead staff and the PBL processes. Although much of this was 
likely down to the skills and abilities of the PBL lead, it shows the potential value of team-working for 
staff support and development. 


For Year 7, about 50% of the timetable was assigned to PBL and this reduced in Year 8. PBL included 
ICT, science, design and technology, and humanities, as well as an element of English. The PBL 
team did not, however, possess the whole range of subject expertise, and where necessary the 
specific departments and teachers from elsewhere in the school contributed to the planning and 
design of the REAL Projects and materials. The team were very cohesive which helped with joint 
planning, and this was helped by having a period when the full team could meet each week on Friday 
afternoons. 


Outcomes 


Perceived outcomes of the intervention 
The pupils’ perspective 


The pupils interviewed talked with enthusiasm about a variety of projects they had worked on, 
developed, and presented to an external audience, including projects about the environment and 
recycling issues, the Romans, health and social care, school improvement issues, reflections on 
inspirational people, and history heroes—among many other topics. 


Pupils thought that PBL work was more relaxing and fun than subjects taught in a more traditional, 
didactic way. They felt they could learn useful knowledge and develop skills that would be valuable in 
real life situations, in career choices, and in their development in life later on. This view is illustrated in 
comments such as: 


‘You feel proud of yourself, it determines you more and more, then one day you’re probably 
going to have the passion to be a historian or someone who knows a lot about geography, so 
it really helps for future’ (School C pupil). 


The PBL environment was also perceived as encouraging all pupils to participate in tasks through 
team working and appropriately differentiated work. Some pupils felt that it helped them develop the 
tools to evaluate their own work and the work of others while developing a better awareness of areas 
that they could improve on. Others mentioned with pride that they developed creativity skills, the 
ability to listen to and accept their peers’ critique and feedback, as well as enhanced research skills 
and an ability to understand and remember the material covered because they found it more 
‘connected’, ‘interesting’ and ‘fun’. 


‘In REAL Projects, [learning is] connected, it’s building up your skills so that you can use them 
in real life situations’ (School F pupils). 


Education Endowment Foundation 45 


Project Based Learning 


[Feedback from peers] was helpful for us so we know what to do next... the first time people 
wrote like good handwriting or good use of punctuation, but we needed more depth. Because 
we then had more capability we started to actually write something that would be useful for 
the person’ (School C pupils). 


Pupils also mentioned that they appreciated their artistic involvement in projects and engagement with 
technology which made their learning more interesting and engaging. Very often their enthusiasm 
motivated them to work more at home. 


‘| sometimes do work at home... it gives me more knowledge in what | am learning about. The 
projects actually motivate us to do more research cause we’re really determined so we want 
to make it the best and so we go home and maybe one person in the group will write the script 
even though Miss said you didn’t need to do this but it will make us really better’ (School C 


pupils). 
The perspective of teachers and PBL leads 


There were a number of success stories that teachers and senior members of staff were keen to 
share in the interviews. All of these related to the confidence the pupils developed through their 
project-based work which was demonstrated through significant improvements in their oracy skills— 
their ability to express themselves fluently and confidently in front of an audience. Oracy skills and 
confidence in communicating were seen as particularly important for the future workplace. 


‘We have a child here [who was] very, very withdrawn, couldn’t make friends easily... but she 
actually [performed] in assembly in front of all the kids. You can’t measure that. And that just 
made such a difference to all of us and even if you’ve had the week from hell, you get in there 
and you celebrate the achievement with the kids and they’re all applauding each other. When 
you see that, you come out thinking “I’m ready for next week now”, and it does uplift you’ 
(School F teachers). 


‘I’ve noticed an improvement in confidence. With lots of them they have become a lot more 
confident in their own ability, but also in taking the initiative a bit more. We've had quite a few 
pupils now making speeches at the exhibitions and that’s really good for confidence and 
public speaking’ (School C, PBL lead). 


‘She’s always been a very high-achieving student and she’s very top set, but | think the thing 
she’s got from PBL is her oracy—she’s much more confident now to speak and to express her 
opinions’ (School B follow-up visit, deputy head). 


Some teachers also expressed views that special educational needs pupils had particularly benefited 
from PBL. SEN pupils had been observed to engage quickly within the PBL environment because of 
the connected information they were presented with. This meant that there was less room for 
disruption in the classroom. 


‘We work with a lot of autistic and ADHD children and it was perfect for them. They weren't 
having what they would see as random information given to them. It was completely aligned 
so they were heading along one path of journey of discovery which was perfect for autistic 
children’ (School D headteacher). 


Teachers felt that the major outcomes of PBL for the pupils were increased engagement and more 
positive attitudes to learning, which were evident in the classroom but also came through in their 
responses in student voice reviews. A key factor that contributed to pupils’ engagement, according to 
the teachers, was their ability to see the purpose of what they were doing and take ownership of their 
learning within an environment that recognised the diverse areas in which pupils can excel. 
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‘The pupils are incredibly positive about their learning, they feel rewarded, they feel like their 
talents are recognised. They feel like they have an opportunity to succeed in a range of 
different areas rather than the very narrow metrics that we use ordinarily. They are much 
more confident, they are noisy but actually incredibly well organised, and have a great attitude 
to learning’ (School F headteacher). 


Improvement in literacy was another outcome mentioned by school staff, particularly in the improved 
quality of pupils’ writing. Moreover, in preparation for the new school year, the desire to implement 
PBL effectively often led to special attention being given to facilitate a smooth transition for pupils to 
the secondary school environment through stronger cooperation with the feeder primary schools. This 
helped secondary schools to be better prepared to meet the learning needs of all pupils: 


‘We REALLY know our pupils in PBL, we really know them back to front and | think they do 
feel a bit safer | suppose, and with a high proportion of statemented kids here, it’s nice and it 
does help that transition | think, give them a bit of consistency’ (School C, PBL lead). 


Furthermore, staff stated that parents’ feedback was very positive about PBL as pupils’ enthusiasm 
with their learning and increased involvement in PBL activities were noticed and appreciated at home: 


‘We had parents’ evening last week and it was quite interesting, a lot of the parents were 
saying how much the kids were enjoying doing it and it was actually feeding back into their 
home lives where you're not sure an English lesson or Chaucer would have had that same 
kind of effect, and | think cause it’s in smaller chunks, the pace of what you are delivering is 
quite intense but we moved on quite quickly and | think that’s been huge’ (School D teacher). 


Schools often used PBL as an opportunity to engage more parents in the school. One school set up a 
system where text messages would be sent to parents to inform them about PBL key events and gave 
pupils the opportunity to continue their PBL work at home by allowing them outside access to Google 
documents. Another school set up Learning Blogs for pupils for them to reflect on their PBL work 
which could be accessed by parents. As a result, some parents felt more engaged with their child’s 
learning. Furthermore, all schools took extra steps to establish stronger home-school links in order to 
encourage more parents to attend public exhibitions as this was perceived particularly valuable. 


‘I’m getting dates in the diary for the exhibition a lot earlier than | was last year... We can let 
parents know sooner and we can put it in the parent bulletin—one thing that’s really helped | 
think, especially for the performance is that we’ve now got a facility in the office to text parents 
and the office text every single Year 7 parent the day before and said don’t forget about the 
performance tomorrow night and | think that really helped’ (follow-up visit, PBL lead). 


Perceived benefits and pupil progress in School E 


In School E, PBL was seen to help departments to think and work together and therefore had positive 
benefits in other subjects and areas of the school, potentially leading to benefits for pupils beyond 
Years 7 and 8. 


It was also found that REAL Projects supported existing primary/secondary transition activities, as well 
as fostering wider local contacts and educational opportunities. For instance, the school had worked 
with a local museum and local designers in some of the projects. Likewise, a number of staff visited 
other PBL schools, both locally and further afield, which, they suggested, helped bring in new ideas. 
More generally, working in a team was found to help many staff reflect on their own practice and skills 
and it was thought it would therefore support staff development more generally. For the pupils’ 
progress through the school it was felt that the PBL approach to learning instilled valuable learning 
skills, such as research skills, as well as giving them an introduction to subject- or department- 
organised teaching, which they would not have been familiar with from primary school. 
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As with many of the other schools there were some challenges in assessing impact. It was found that 
the approach in the first year of delivery lacked consistency making it difficult to draw firm conclusions 
about progress. However, in the second year, the systems were more aligned and integrated with the 
whole-school approach allowing direct comparisons to be made, both year on year and between 
subjects. The school emphasised that consistency in assessment and the ability to compare progress 
was important for maximising the returns from PBL, as well as giving wider credibility to the approach. 


In general, feedback from the various departments suggested that the Year 7 group as a whole had 
made more progress than previous years. It was also felt that PBL encouraged deeper leaning of 
particular topics and increased the breadth of content that could be explored by going beyond the 
national curriculum. Specifically, it was felt PBL helped the development of a wide range of skills for 
the future which would be useful throughout the pupils’ education, such as research, team work, and 
presentational skills. The school had also found the PBL approach more suited to low-ability learners 
and was able to point to instances of far more progress being made by such learning than would have 
been expected. However, it did recognise this was not in itself a solution for all such pupils and that 
more planning and adaption of lessons needed to be done. 


Overall, this school was very enthusiastic about PBL and felt that it was of substantial benefit to its 
teaching and learning. It was also positive about the potential benefits to other schools, however it did 
point to the need for the school to be in a ‘secure’ position, both internally and externally. From its 
experience, a great deal of commitment had to be made to properly implement the project, and to 
reap the benefits for pupils. 


Perceived benefits and pupil progress in School F 


For School F, the most important tangible benefit was improved engagement in the learning process, 
for both pupils and their parents. With regards to parents, we were told that far more were attending 
parents evenings than had previously and more were also accessing on-line reports. 


The school did not yet have the data to directly assign academic impacts to PBL, however it felt that 
by improving engagement—even in just one target group, such as white working-class boys—it was 
likely to improve academic performance. The school also pointed to future learning benefits, including 
life-long learning and skills valued by employers. It was felt that embedding these skills at this early 
stage would have the benefit of encouraging more pupils to continue their education by doing IBs or 
more vocational courses. We were told of improvements in terms of PBL directly supporting the 
primary/secondary transition as well as the longer-term integration of the Year 7 pupils into the school. 


We were also told that the programme is likely to go on to help support local employment needs, for 
example in the area of STEM and progression to Higher Education. It was also evident that the 
programme was playing a valuable role in terms of staff training and support, in particular through 
team working, and, more generally, for the school to help meet its broad educational aims. 


Formative findings 


How the intervention can be improved 


The following suggestions refer to process evaluation findings that might be considered by the 
Innovation Unit and addressed in their training with the schools. The training and support offered was 
perceived to be excellent with ample help and support offered along the way. However, if schools are 
aware of the suggestions that follow, there will be greater potential for the intervention to be delivered 
more effectively. For example, if schools are actively encouraged to share best practice with other 
schools from the beginning of the project and are able to build this into their timetable, then better 
decisions might be made regarding its implementation. Likewise, if schools are alerted to the danger 
of project-based work being occasionally approached on a more superficial level by pupils, this would 
raise understanding about the importance of closely supporting and monitoring pupils’ work and 
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progress. Last but not least, embedding PBL across the school seems to be a useful target for 
schools to be aiming towards. 


Student choice, but ensuring depth of learning 


The student interviews revealed that students would like more of a say in what projects were 
undertaken, and it was felt that this would lead to a higher level of participation and engagement. This 
suggestion was welcomed by some schools, however more student choice does not mean that 
teachers would not need to monitor pupils’ decisions in order to challenge and extend them. On the 
contrary, they would need to work closely with the pupils to ensure that they are engaged into a 
reflective, deeper approach to learning rather than getting side-tracked into the more superficial or 
presentational aspects of the project, as illustrated in the quote that follows: 


‘I think it’s that depth now that we really need to be looking at. For example, | have one girl 
that worked extremely hard at the last ‘Trashion Show’ project, bearing in mind the whole 
scheme of learning was about 6Rs, sustainability, consumerism, the effects that consumerism 
is having on the planet and then she made this product that was tennis balls that held things... 
she'd done a good job of getting donated tennis balls but it was like a useless item, | felt that 
she’s missed the point of the project and | tried in lots of occasions to give a critique to move 
her forward but she was adamant this was what she wanted to do. ...it’s that depth and in her 
evaluation she has picked up on it but to the point that the penny’s dropped’ (School F, follow- 
up meeting, PBL lead). 


Embedding PBL across the school 


When this deeper learning becomes better embedded within the PBL environment, and the 
organisational challenges fully addressed, then it will be more feasible for schools to develop a more 
holistic vision about the potential benefits of this type of learning. For example, teachers not already 
involved in PBL may start to see benefits of PBL on pupils moving up through the school, especially in 
terms of confidence and skills, and their deeper knowledge. Encouraging teachers not involved in PBL 
to observe learning in PBL classrooms might also help them to see how it could work for them. A 
‘cultural change’ might then start to take place in schools such that more teachers adopt project-based 
learning elements in their teaching. If this change occurs, schools may move towards more of a 
common shared vision and more interdisciplinary working. Comments that reflect the need for such a 
school-wide interaction include: 


‘Within the next year, we aim to introduce [PBL to] more subjects, we do need some structure 
to create that cultural change; it’s got to work holistically, so it will need that structure at the 
early stages’ (School B, follow-up visit, deputy head). 


‘It’s the investment with staff—if we could roll it out to every curriculum area, to every group, | 
think it would be different and that would be the long term goal’ (School A headteacher). 


‘She’s working with Maths and Science team, so it’s trying to roll out our core ethos into other 
areas so it’s not as directed from the REAL team, it’s how it fits into the rest of the school’ 
(School F, follow-up meeting, PBL lead). 


Sharing best practice across schools 


Sharing ideas with other local schools and becoming familiar with best practice in schools where PBL 
has been implemented successfully was perceived as being a useful mechanism for continuous 
professional development and as a source of inspiration. In addition, resource-sharing through the 
online community—through websites and Google handouts—facilitated communication and the 
development of ideas around PBL. It was also mentioned that the enhanced communication among 
schools could be even more valuable in the long run than the initial support from the Innovation Unit. 
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However, many schools found it difficult to engage with other schools (even locally) due to time 
pressures in everyday school life or because of extra demands placed upon teachers higher up the 
school. 


‘| think we would have loved to have gone to the exhibitions of some of the other school 
projects... [but] everything becomes so busy doesn‘ it’ (School A headteacher). 


‘| think if we’ve got a team of schools around us that are all working towards the same thing... 
that’s probably more important than it is having somebody come from the Innovation Unit 
individually. [...] | work with colleagues from other schools around here and we share ideas 
and we work together, so I’d kind of want that relationship moving forward’ (School D, PBL 
lead). 
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1. Adopting PBL had no clear impact on either literacy (as measured by the Progress in English 
assessment) or student engagement with school and learning. 


2. The impact evaluation indicated that PBL may have had a negative impact on the literacy 
attainment of pupils entitled to free school meals. However, as no negative impact was found 
for low-attaining pupils, considerable caution should be applied to this finding. 


3. The amount of data lost from the project (schools dropping out and loss to follow-up) 
particularly from the intervention schools, as well as the adoption of PBL or similar approaches 
by a number of control group schools, further limits the strength of any impact finding. 


4. From our observations and feedback from schools, we found that PBL was considered to be 
worthwhile and may enhance pupils' skills including oracy, communication, teamwork, and 
self-directed study skills. 


5. PBL was generally delivered with fidelity but requires substantial management support and 
organisational change. The Innovation Unit training and support programme for teachers and 
school leadership was found to be effective in supporting this intervention. 


Interpretation 


The impact evaluation indicates that the Learning through REAL Projects intervention does not 
improve literacy or engagement outcomes for participants overall, in the intervention group compared 
to the control group, and may have had a negative impact on pupils eligible for FSM, however major 
limitations to the trial add uncertainty as to the validity of these results. In this interpretation we 
discuss how the impact findings should be interpreted, as well as some of the broader issues arising 
from this project. In addition we highlight the useful lessons from this project for evaluating complex 
interventions requiring large-scale school-level change. 


We summarise and consider the interpretation under five headings, including: academic impact, 
impact on engagement, process, and large-school change interventions. 


Academic impact 


Overall the project did not find that PBL had an impact on literacy for all of the pupils in the 
intervention group compared to those in the control group. The analysis did show a statistically 
significant negative finding for FSM pupils in the intervention group (in other words, which suggested 
they did worse as a result of doing PBL). However, these findings should be interpreted with caution. 
This particularly so in regards to the comparison group—what the counterfactual to schools doing PBL 
was, the group imbalance after randomisation, and the level of attrition from the intervention group 
(47%) (see limitation section below) as well as the fact that it included a reasonably small number of 
schools and therefore was lacking power. 


In terms of the counterfactual, a number of the control schools were actively using PBL approaches. 
Related to this, it is important to be aware that schools were recruited on the basis that they had the 
ability and capacity to make the substantial organisational changes required to implement PBL, as 
well as the incentive and desire to make improvements in how they delivered the KS3 curriculum. In 
addition, all schools recruited were willing to pay £10,000 towards professional development to 
support the implementation and delivery of PBL. It seems reasonable therefore that the control 
schools were not simply passive (or engaged in ‘business as usual’) during the trial period, but rather 
sought other ways to improve their KS3 teaching (including using PBL). The process evaluation found 
that some control schools delivered some project based learning and that four control schools used a 
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variation of PBL to teach at least 20% of their Year 7 curriculum, sometimes with bought-in additional 
support. This makes it difficult to assess the impact of Learning through REAL Projects over and 
above the impact of the PBL interventions delivered by the control schools. 


It would also seem credible that reducing the trial period from two years to one year limited the time 
available for the intervention to become embedded in a school, therefore reducing the ability to 
demonstrate an impact. 


The significant negative finding on literacy outcomes for FSM pupils is, however, of concern. It is 
particularly difficult to interpret given that the same negative impact was not found for low-achieving 
pupils. It is possible that the Learning through REAL Projects intervention is not as accessible to FSM 
pupils as for other pupils. The autonomy provided by projects and the requirement for more 
independent self-directed study may require skills that are less developed by these pupils, or it may 
be that access to resources (such as computer or internet access) is not as available outside of 
school for disadvantaged pupils. The process evaluation, however, did not provide any evidence to 
support this hypothesis. This element of the evaluation was based on classroom observations and 
feedback from headteachers, project leads, teachers, and pupils in the schools. It found that PBL was 
well received and considered to be of benefit in terms of developing oracy, communication, team 
working, and research skills. The intervention was also perceived as being of benefit to low-attaining 
pupils more generally. 


It is worth noting that although literacy attainment was chosen as the primary outcome for this study— 
given the primary purpose of the EEF—it was not the principal aim of the Learning through REAL 
Projects intervention. Its key aims were to promote deeper learning of topics, learning skills, and skills 
relevant to future work and study, as well as interest and engagement in learning more generally; 
literacy skills only formed a part of the picture. Nonetheless, there was a clear rationale as to why 
there could have been a positive impact on literacy, and accordingly the funder, developer, and 
evaluator were keen to investigate whether a PBL curriculum can be implemented which can promote 
literacy attainment to at least the same extent as a more traditional teaching approach. However, the 
conclusion from this trial has to be that adopting PBL had no clear impact on either literacy (as 
measured by the Progress in English assessment) or student engagement with school and learning. 


Impact on engagement 


As with the academic outcomes, we did not find any evidence that the intervention impacted either 
positively or negatively on engagement with school or learning from the attitudes survey or the 
attendance data. However, the research limitations regarding the control group and attrition, described 
above for the academic outcomes, also apply to these outcomes and limit the conclusions that can be 
drawn from the data. 


It should, though, be noted that the underlying aims of this PBL intervention are long-term benefits in 
terms of softer secondary outcomes—such as engagement with learning as well as skills for work and 
later study—and it may be the case that there would be differences between the two groups at a later 
date, such as when accessing further and higher education or employment. It may, however, be that 
our measures of these softer skills of engagement and school attendance were not the most 
appropriate measures to use in this study and that measures of the skills valued by employers and 
those which support further learning, as well as later outcomes, should be the focus of future study. 
This trial was only able to measure impact after one year of implementation when schools were just 
beginning to understand how to develop effective projects and getting used to a new way of teaching. 
A longer time for this practice to embed in schools may also lead to more positive outcomes. 


Implementation and large school change interventions 
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The process evaluation tells a much more positive story compared to the impact evaluation findings. 
There was a great deal of enthusiasm for PBL from the intervention schools that completed the first 
year, as well as for those that continued into the second year. Almost without exception, those schools 
reported finding positive benefits from the programme in terms of attainment, confidence, learning 
skills, and engagement in class. While there was a high level of attrition from the intervention group 
schools, this was not generally due to schools having difficulty with implementing the intervention but 
to other factors in the school including leadership changes and academisation (with the exception of 
two schools where the intervention affected their decision to continue). 


We found from our observation visits, and feedback from the Innovation Unit coaches, that the 
intervention was implemented with a high degree of fidelity over the one year research period. The 
level of support provided by the Innovation Unit model, both from Classroom and Leadership 
Coaches, was described as essential to supporting this fidelity and for successful delivery of the 
intervention in schools. Support from school leadership and a strong REAL Projects team who are 
bought-in to the principles of PBL are also key to successfully implementing this intervention. Almost 
all schools reported that assessment of REAL Projects, in a way that could be used to show progress 
for accountability purposes, was a major challenge to its implementation. 


Effectively implementing PBL at a whole school or cohort level is challenging due to the necessary 
degree of organisational change. It should of course be noted that all of the schools which 
implemented PBL were both willing and generally had the capacity for such change. We found, 
however, that the Innovation Unit’s Learning through REAL Projects implementation processes were 
particularly effective for the target (that is, willing and with the capacity) schools, and the feedback 
from those schools was almost entirely positive. 


The key elements were to support both the classroom practice with expert coaches, as well as provide 
support to the heads and leadership teams. The latter varied considerably, largely depending on the 
particular contexts of the schools, and is therefore difficult to describe in a generalised way. The 
Leadership Coach worked on a case-by-case basis providing support as was necessary at different 
levels depending on the requirements of the school. However, such a skill is largely generic and it 
may be possible for this to be provided in other ways than a specific PBL coach. The Classroom 
Coaches on the other hand had to have very specific skills and be experienced in the delivery of 
projects in the classroom, and their knowledge had to be imparted in a fairly consistent way across a 
wide range of schools. 


The implementation was also effectively supported through high quality initial and centralised training 
(QUEST) which besides providing schools with the necessary skills and materials, allowed schools to 
feedback and discuss their own situations and potential issues with both the Innovation Unit team and 
colleagues from other schools. This was a two-way learning process which was particularly evident 
from the improvements made between the pilot QUEST and subsequent first round QUEST training, a 
process essential for the implementation of a complex intervention such as PBL. In addition, the 
ongoing support through newsletters, online community pages, and blogs also helped with the 
implementation. 


Limitations 


The key design limitations were the validity of the counterfactual, moderately high levels of missing 
data, imbalance between intervention and control groups at the pupil level after attrition, and the 
limited time the intervention was given to embed in schools. 


The issue of the validity of the counterfactual occurred through not being able to isolate the 
intervention and control groups as a third of control group schools were doing a variation of PBL. 
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The level of buy-in required from schools signing up to the project, including the monetary 
commitment, led to only schools very motivated to implement PBL signing up to the project. However, 
for some that did not get allocated to the intervention group, faithfully adopting the control condition 
was seen as being detrimental to their pupils’ learning and therefore some of these schools chose to 
implement a version of PBL anyway. This creates a substantial challenge for an RCT design with such 
an intervention and consideration could be given to alternative designs such as time-series for future 
similar evaluations. 


There was also a high level of attrition in collecting the assessment data during the project, particularly 
from intervention group schools as well as issues with missing baseline data. It was not possible to 
collect data from five intervention schools in this project with two withdrawing after randomisation (due 
to no longer wishing to do the intervention) but before any intervention implementation, two 
withdrawing towards the end of the project (due to their perceptions of the intervention) and one being 
unable to secure parental consent for the assessment. The analysis on an intention-to-treat basis will 
therefore be biased to some degree, with a large chunk of data missing, not at random, from the 
intervention group. Those that withdrew were less likely to have implemented the intervention and 
therefore the analysis may be positively skewed if the intervention does have an impact, however the 
level of missing data introduces unknown bias and reduces the power of the study to detect an effect. 


We also need to be aware of the limitations of what was being measured, and whether the measures 
were likely to be affected by doing the intervention. Literacy as assessed by the PiE assessment was 
chosen as the primary outcome because it was judged to be the EEF academic assessment most 
closely aligned with what PBL was seeking to do: improving the pupils’ literacy skills. The assessment 
chosen, however, only measured a select set of literacy outcomes and in particular did not include any 
extended writing tasks which might have been more affected by the intervention. As mentioned above, 
literacy was not the primary aim of the Learning through REAL Projects intervention and therefore it is 
possible that a different assessment looking at other factors more aligned to the intervention would 
have found a significant effect (positive or negative). 


Related to the above is the time scale or period allowed for the intervention to have an impact, in 
practice the time between the intervention starting and the outcome being tested (as well as the time 
delay before the control group can access the intervention). The reduction from a two year trial to a 
one year trial could have reduced the impact that could be detected and if we had measured 
outcomes after two years we may have been more likely to see a difference in outcomes between the 
control and intervention group. Also, some of the predicted outcomes for PBL relate to future 
education phases and life-long learning, the effects of which cannot be seen until later, and therefore 
not possible to measure in this project. 


Given that it is not possible to conclude with any confidence that PBL had a positive or negative 
impact on literacy outcomes we are limited in what we can say about the generalizability of using PBL 
to improve literacy and engagement more widely in other schools. The feedback from the process 
evaluation may be relevant for most schools seeking to broaden their curriculum, however, in terms of 
the messages about the organisational change required to implement PBL, the learning can only be 
really generalized to those schools that are willing and that have the capacity to manage the change 
process. We cannot say anything about the likely impact if such an intervention were to be imposed 
on a school. 


Future research and publications 


The need for improving skills appropriate for further study and those valued by employers in the 
modern workplace (a central aim of PBL) has not diminished, but probably increased. This study 
picked up the value of these skills to pupils’ learning and future potential through the process 
evaluation, but was not able to measure these skills as an outcome. Further research should be 
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commissioned as to the ways in which these skills can be measured, and to how schools can play 
their part in providing young people with such skills and ways in which they can be measured. 


For future evaluations of similar interventions involving significant investment from schools and large 
scale organisational changes, careful consideration needs to be made about whether randomised 
controlled trial designs are appropriate to assess effectiveness in light of the difficulties faced during 
this project. Other rigorous research designs may be better suited to this style of evaluation and 
particularly those that allow a longer term (five- to ten-year) view to be taken. 


The research team do intend to publish a number of papers from this evaluation and currently an 
extended version of the literature review is due to be published shortly in a journal. In addition, the 
team are advising the KATA foundation in Denmark on a PBL project that will promote STEM subjects 
and the impact will be assessed over a long period of time (up to ten years). 
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Appendix A: EEF cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention 
over three years. More information about the EEF’s approach to cost evaluation can be found here. 
Cost ratings are awarded as follows: 


Cost rating Description 


ELEEE Very low: less than £80 per pupil per year. 
EELEE Low: up to about £200 per pupil per year. 


ELEELE Moderate: up to about £700 per pupil per year. 
ELEEEL High: up to £1,200 per pupil per year. 
ELEELEE Very high: over £1,200 per pupil per year. 
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Appendix B: Project-based learning: a review of the 
literature 


Defining characteristics of project-based learning 


Project-based learning is a student-centred form of instruction which is based on three constructivist 
principles: learning is context-specific, learners are involved actively in the learning process and they 
achieve their goals through social interactions and the sharing of knowledge and understanding 
(Cocco, 2006). It is considered to be a particular type of inquiry-based learning where the context of 
learning is provided through authentic questions and problems within real-world practices (Al-Balushi 
and Al-Aamri, 2014) that lead to meaningful learning experiences (Wurdinger, Haar, Hugg and Bezon, 
2007). Blumenfeld, Fishman, Krajcik, Marx and Soloway (2000), for example, described the process 
of project-based science as follows: 


‘The presumption is that students need opportunities to construct knowledge by solving real 
problems through asking and refining questions, designing and conducting investigations, 
gathering, analysing, and interpreting information and data, drawing conclusions, and 
reporting findings’ (p.150). 


It has been argued that the freedom and challenge that pupils experience as a result of solving the 
problems that arise in designing and building their projects result in high levels of student engagement 
(Wurdinger et a/, 2007) due to the cognitive challenge as well as the strong affective, ethical and 
aesthetic dimensions that form part of a well-designed project (Wrigley, 2007). Thomas (2000) 
identified five essential characteristics of projects: 1. Centrality, 2. Driving question, 3. Constructive 
investigations, 4. Autonomy and 5. Realism, with the importance of student collaboration, reflection, 
redrafting, and presentations emphasised in other publications (Kwon, Warderip and Gomez, 2014; 
Patton, 2012). The uniqueness of project-based learning is the construction of an end product, a 
‘concrete artefact’ (Helle eft a/., 2006) which represents pupils’ new understandings, knowledge and 
attitudes regarding the issue under investigation often presented using videos, photographs, sketches, 
reports, models and other collected artefacts (Holubova, 2008). 


It is argued that it can help foster self-regulated learning and can promote pupils’ conceptual 
knowledge within a systematic process of documenting and reflecting on learning (Barak, 2012). 
Pupils learn to be self-reliant through goal-setting, planning and organisation, they develop 
collaboration skills through social learning and become intrinsically motivated by being encouraged to 
exercise an element of choice while learning at their own level (Bell, 2010). Project-based learning 
has been explored in various contexts and in different phases of schooling ranging from the early 
stages of education through primary and secondary school to higher education. 


Overview of the evidence for the effectiveness of project-based learning 


Most of the reviewed studies did not involve random allocation of participants to control and 
experimental groups and, as a result, a causal link between project-based learning instruction and 
positive student outcomes cannot be established with certainty. The majority of these studies were 
based on a quasi-experimental pretest-posttest design with some baseline equivalence established 
for the outcomes measured at the classroom level. Some studies of weaker quality were based on 
observations of pupils’ behaviour, attitudes and accomplishments in a project-based learning 
environment without the presence of a comparator group (for example, Barak and Asad, 2012; 
ChanLin, 2008; Cuevas, Lee, Hart and Deaktor, 2005; Morales, Bang and Andre, 2013). Other studies 
have used state standardised test averages against which to compare the performance of 7"/8'" grade 
pupils (Geier et a/., 2008) and 12" grade students (Schneider, Krajcik, Marx and Soloway, 2002). 
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Sweller, Kirschner and Clark (2007) have emphasised the importance of randomised controlled 
experimental studies of different instructional procedures to provide stronger and more reliable 
evidence on the effectiveness of project-based learning. 


Pre-school and primary school 


Implementation of a project-based concept mapping developmental programme to facilitate children’s 
experiential reasoning and comprehension of relations (Habok, 2015) reported positive results for the 
experimental group that attended one of the two kindergartens in Hungary. In particular, even though 
the experimental group started with a disadvantage in achievement, there was a significant increase 
in this group’s development compared to the control group. Habok concluded that the use of concept 
maps in school practice holds promise as a visual expression tool in promoting understanding of 
connections and causalities. Another study with pre-school science teachers in Sweden (Ljung-Djarf, 
Magnusson and Peterson, 2014) argued that a learning study project model (a kind of action research 
that combines variation theory with the concept of lesson study) has the potential to promote pre- 
school science. 


In their quasi-experimental study on the effectiveness of project-based learning in primary school in 
Greece, Kaldi, Filippatou and Govaris (2011) argued that primary age pupils can develop content 
knowledge and group work skills in addition to motivation and positive attitudes towards peers from a 
different ethnic background through project based-learning instruction. Similarly, Karagalli and Korur 
(2014) conducted a quasi-experimental study in Turkey with fourth-grade science pupils (equivalent to 
Year 5 in the U.K.) and found a statistically significant effect in terms of academic achievement and 
retention of knowledge for the project-based learning pupils. A U.S. study that explored the 
effectiveness of a project-based approach in 2" grade (equivalent to Year 1 in the U.K.) social studies 
and content area literacy (Halvorsen, Duke, Brugar, Berka and Brown, 2012) reported positive 
outcomes for low-SES pupils and claimed that the project-based learning approach has the potential 
to help narrow the gap between low and high-SES pupils in social studies and literacy for 24 grade 
pupils. The study employed a ‘design or formative experiment approach’ (p.10) where six teachers 
and a subset of their pupils participated in the study. Two teachers were from high-SES schools and 
four teachers from low-SES schools. The teachers in the low-SES schools implemented project-based 
units in their teaching which were developed by the researchers. In addition to student assessments, 
data were also collected through classroom observations and teacher interviews. The study had a 
number of limitations, such as a small sample size (N=10-12 from each class with 43 children in low- 
SES and 20 children in high-SES classrooms), lack of a control group and researcher designed 
assessment measures that may be less reliable and valid in comparison to other published 
standardised measures. 


Secondary school 


Al-Balushi and Al-Aamri (2014) conducted a quasi-experimental study with 62 11' grade female pupils 
(equivalent to Year 12 in the U.K.) in Oman that explored the effect of environmental science projects 
on pupils’ environmental knowledge and attitudes towards science. Two classes were randomly 
assigned into an experimental group and a control group. The findings were positive with the 
experimental group significantly outperforming the control group in the Environmental Knowledge Test 
and the Science Attitudes Survey. The authors acknowledged, however, that a novelty effect could not 
be ruled out as pupils’ enthusiasm in the experimental group in using new technology to design their 
products could have led to the more positive results in the post-tests. 


In history learning, Hernandez-Ramos and De La Paz (2009) had eighth grade pupils in the U.S. 
(equivalent to Year 9 in the U.K.) learn to create multimedia mini-documentaries in a six-week history 
unit. Compared to pupils who received traditional instruction, pupils that engaged in the project-based 
learning curriculum demonstrated positive affective benefits and significant gains in content 
knowledge as well as historical thinking skills. This was a quasi-experimental study using a pretest- 
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postttest design and there was no random allocation of pupils or teachers to control and experimental 
conditions. Therefore, it cannot be inferred with certainty that the knowledge gains are necessarily the 
result of technology-enhanced project-based learning at the intervention school as other teaching and 
learning activities could have contributed to the positive results. 


Another quasi-experimental study carried out in the U.S. (Hsu, Van Dyke, Chen and Smith, 2015) 
explored seventh graders’ (equivalent to Year 8 in the U.K.) development of argumentation skills and 
construction of science knowledge in a graph-oriented computer-assisted project-based learning 
environment. A significant difference in science knowledge, counterargument and rebuttal skills was 
found in favour of the treatment condition. In another U.S. study, Geier et a/. (2008) reported that 7" 
and 8" grade pupils that participated in project-based inquiry science units showed increased science 
content understanding, better process skills and significantly higher pass rates on the statewide test 
over the remainder of the district population. 


Boaler (1998) conducted a longitudinal study of mathematics instruction comparing an open, project- 
based environment to a traditional approach and it followed two cohorts of pupils in two British 
secondary schools from Year 9 to Year 11. Even though this study did not involve the random 
allocation of participants, it employed a closely-matched control group in terms of socioeconomic 
status, prior mathematics instruction and attainment. A variety of instruments were used to measure 
pupils’ skills, attitudes and attainment. The main finding was that the two groups developed different 
forms of knowledge. The pupils learning mathematics in the project-based environment developed 
conceptual understanding which often required creative and deeper thinking in contrast to the 
procedural knowledge acquired by the traditional instruction group which was mainly based on 
information recall. In addition, more pupils at the project-based school succeeded in passing the 
General Certificate of Secondary Education (GCSE) at the end of the three-year study than those 
pupils receiving the traditional instruction. 


Other studies have shown higher learner motivation in a project-based learning environment with 
fourteen and fifteen year old girls in Israel showing increased interest in learning scientific- 
technological subjects (Barak and Asad, 2012). Project-based learning as related to STEM (science, 
technology, engineering and mathematics) curriculum design for female senior high school pupils in 
Taiwan led to gains in terms of enjoyment, engagement with the project and the ability to combine 
theory and practice effectively (Lou, Innovation Unit, Shin and Tseng, 2011). This study was an in- 
depth investigation of 84 pupils’ cognition, behavioural intentions and attitudes in the project-based 
STEM environment and involved text analysis and questionnaire survey as the main data collection 
tools. 


The 10-11 year old pupils in ChanLin’s (2008) qualitative study in Taiwan developed skills in 
synthesising and elaborating knowledge and in engaging in scientific exploratory tasks with the use of 
technology. Project-based learning has also been explored as a method of instruction with low- 
achieving pupils in Israel (Doppelt, 2003) and the U.S. (Cuevas et al., 2005), and with second chance 
school pupils in Greece (Koutrouba and Karageorgou, 2013) with positive outcomes. Doppelt (2003) 
found that scientific-technological project-based learning helped improve low-achieving pupils’ 
motivation and self-image by allowing pupils to succeed early on in the process and led to more pupils 
achieving the college admittance requirements. Doppelt’s study was a field research project that used 
qualitative and quantitative tools (portfolio analysis, observations, interviews, matriculation 
examination results and assessment of pupils’ projects) with a sample of 54 10" to 12" grade pupils 
(fifteen to eighteen years old). 


Encouraging results were also reported with high school high achievers in Israel where 60 pupils from 
three experimental classes in comprehensive high schools exhibited a significant increase in formal 
technological knowledge and skills and more positive attitudes towards technology in comparison to 
the pupils in the three control classes which were drawn from technological high schools (Mioduser 
and Betzer, 2007). However, the different type of schools involved suggests differences in student 
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take-up and characteristics, and indicate an unequal student comparison which limits the strength of 
the findings. Some studies have shown mixed results. For example, in their quasi-experimental study 
with 13 year old children (grade 8) taking computer courses in Greece, Boubouka and Papanikolaou 
(2013) found no significant effect of project-based learning on student achievement but a statistically 
positive effect on self-perceived learning performances. 


Project-based learning studies in higher education and in pre-service teacher 
training 


A number of studies have explored the effectiveness of project-based learning in higher education in 
different countries. Most of these studies have focussed on engineering education. For example, 
Ruikar and Demian (2013) made links with industry engagement through multimedia podcasting in the 
U.K., Hassan and his colleagues (2008) adopted an integrated, multicourse, project-based learning 
methodology in electronic engineering in Spain and Fernandes et al. (2014) followed the project-led 
education model developed by Powell and Weenk (2003), to engage pupils in learning at a University 
in Portugal. In Australia, Stewart (2007) investigated the link between self-directed learning readiness 
and project-based learning outcomes in a postgraduate management course and found that self- 
directed learning readiness, such as having high self-management skills, was a key enabler for 
achievement learning outcomes from project-based learning. Another study (Gibbes and Carson, 
2014) investigated project-based language learning using Activity Theory in a university language 
programme in Ireland. This study reported mixed results in learning outcomes for the study 
participants because of contradictions found in the activity system (e.g. inequitable divisions of labour, 
perceived lack of time due to community obligations or opposition to the rules governing the activity in 
the modules). 


Some studies have applied the principles of project-based learning with pre-service teachers and 
claimed that student-teachers can become better problem-solvers (Mettas and Constantinou, 2008), 
can gain benefits from formative assessment (Frank and Barzilai, 2002) and become more aware of 
the object of learning which can then lead to enhanced learning among pre-school children (Ljung- 
Djarf, Magnusson and Peterson, 2014). 


The review of the literature indicated certain factors that can help facilitate the adoption of project- 
based teaching instruction in the classroom. These are summarised in the section that follows. 


Facilitating factors in the implementation of project-based learning instruction 


On the basis of their study and findings, Al-Balushi and Al-Aamri (2014) concluded that project-based 
instruction is not more demanding than traditional instruction in terms of resources and time and can 
be implemented with few resources, inside the school building and within the time allocated for the 
study of particular topics. 


Modern digital technology is a major enabler for pupils to comfortably engage with the process of 
designing and developing their project as they can document the whole process and easily share their 
creations in a digital format (Patton, 2012). Effective use of technology as an integrated part of the 
pedagogical processes has been found to help both weakly and strongly performing pupils construct 
knowledge in the project-based learning environment (Erstad, 2002). However, Bell (2010) points out 
that children need to be guided and supported in using technology safely and effectively to gain the 
creativity affordances that technological involvement can offer. 


Furthermore, group processes of high quality (conceptualised as group members showing positive 
interdependence, individual accountability, equal participation and social skills) have been found to 
play a pivotal role to the success of collaboration in project-based learning (Cheng, Lam and Chan, 
2008). High quality group work becomes even more important when challenges associated with social 
class differences, gender and attainment hierarchies have been found to affect power relations among 
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some pupils in the project-based learning group leading to unequal learning possibilities with some 
pupils enjoying more agency than others (Crossouard, 2012). Crossouard argues that teachers need 
to be better supported, both within initial teacher education and continuing professional development, 
to develop more sensitivity towards the social and gendered hierarchies that can often be implicit in 
pupils’ discourse, particularly in relation to peer assessment interactions. Issues of social equity can 
thus become part of the pedagogic focus and the language used in the classroom in order to explore 
social relations. 


The successful implementation of project-based learning in the classroom lies on the teacher's ability 
to effectively scaffold pupils’ learning, motivate, support and guide them along the way. Effective 
scaffolded instruction within high-quality experiences will help reduce pupils’ ‘cognitive load’ (Hmelo- 
Silver, Duncan and Chinn, 2007), will enable them to make small successful steps and ultimately 
achieve ‘cognitive growth just beyond their reach’ (Bell, 2010, p.41). Leaving scope for learner control 
of the learning process is crucial with teachers and pupils having to work together to reflect upon the 
purpose of the project, set clear and realistic goals, and make decisions regarding the pace, 
sequencing and content of learning (Helle et a/., 2006). In scaffolding pupils’ learning, teachers may 
need to give pupils insight into the content of the desired response in project-based learning in order 
to allow them to recognise and take up the learning opportunities afforded in the classroom (Gresalfi, 
Barnes and Cross, 2012). Based on their case study findings in the U.S., Grant and Branch (2005) 
concluded that the exploration of cross-disciplinary units and team teaching should be emphasised so 
that pupils can understand how their abilities can be used across domains and avoid the 
fragmentation of skills and knowledge. 


The level of support that teachers get from the school’s senior management (Erstad, 2002) and from 
other colleagues is of particular importance. Lam, Cheng and Choy (2010) concluded that when 
teachers felt well supported by their schools in terms of their competence and autonomy, they were 
more motivated to implement and persist in using project-based learning. 


The use of a two-phase project-based approach has been put forth in the literature as an effective 
approach to first help the pupils become sufficiently competent by developing the knowledge and skills 
needed to then be able to design and make products independently in the second phase (see, for 
example, Drain, 2010; Good and Jarvenin, 2007). Drain (2010) used the Cognitive Apprenticeship 
framework which, on the basis of situated cognition theory, claims that learning is maximised when it 
occurs in real life contexts and pupils engage with authentic problems. This was a case study of a 
primary school class (Year 5) in New Zealand and their teacher during a technology unit. The first part 
of the unit aimed to help pupils develop knowledge of technological concepts and procedures through 
appropriate activities while the second half enabled pupils to be creative and exercise initiative in 
designing and creating their projects. The importance of balancing didactic instruction with in-depth 
inquiry methods has also been emphasised by Grant and Branch (2005). Student assessment needs 
to be aligned to the unique features of the project-based learning process and outcomes with teachers 
identifying suitable assessment moments where they can first generate ‘teachable moments’ 
(Lehman, George, Buchanan and Rush, 2006) and then create formative scaffolds to guide and 
support their pupils along the project process (Hmelo-Silver et a/., 2007). Assessment in project-based 
learning has been described as ‘authentic’ (Bell, 2010, p.43) which, in addition to measuring a child’s 
performance via rubrics, it primarily focuses on reflection, self and peer evaluation. Self-assessment 
skills can help pupils learn to regulate their own learning and acquire ownership of the learning 
process (Ertmer and Simons, 2005). 


How teachers can support project-based learning in the classroom —- what the 
evidence shows 


Mergendoller and Thomas (2005) interviewed twelve expert teachers in project-based learning in the 
U.S. to elicit the teachers’ strategies for implementing and managing the project, and maximizing its 
success. These teachers were recognised as experts within the national PBL community, they had 
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trained other teachers and had made presentations on project-based learning at various professional 
conferences and workshops. Forty three questions formed part of the semi-structured interview 
schedule and covered aspects of overall planning and project planning, carrying out the project and 
the future of project work in the classroom. The interview transcripts were coded into narrative 
segments that led to themes about aspects of project implementation such as time management, 
getting started and managing student groups. This analysis revealed a number of successful 
techniques employed by expert teachers in project-based learning and were grouped around seven 
overarching themes and 18 sub-themes. Each sub-theme comprised a number of principles or 
guidelines which aim to provide practical advice to teachers and are summarised below under each 
theme. 


1. Time management — This theme relates to scheduling projects effectively by coordinating 
project schedules with other teachers, for example, or use block scheduling to increase 
flexibility, and be able to hold to timelines by building in a 20% overrun when planning a 
project or learning when to enforce and when to extend a time line. 

2. Getting started — This theme is about orienting pupils, i.e. getting them think about the 
project well before they begin, giving them a rubric that clearly explains what they are 
expected to search for and try to accomplish and jointly agreeing on grading criteria 
before the start of the project. The ‘getting started’ theme is also about encouraging 
thoughtful work early on in the project in developing a research plan and a suitable 
research question while facilitating a sense of mission. 

3. Establishing a culture that stresses student self-management — Here, responsibility is 
shifted from the teacher to pupils where pupils are involved in project design, they make 
decisions for themselves and they are encouraged to learn how to learn. 

4. Managing student groups — The emphasis is on establishing the appropriate grouping 
pattern, promoting full participation and keeping track of each group’s progress through 
discussion, monitoring and recording evidence of progress. 

5. Working with others outside the classroom, such as other teachers, parents and people 
from the community in order to work out the feasibility and nature of external 
partnerships. 

6. Getting the most out of technological resources, such as judging the suitability of using 
technology for the project, making efficient use of the internet by being encouraged to 
make informed choices in exploring relevant web sites and developing critical thinking 
skills. 

7. Assessing pupils and evaluating projects — This final theme refers, firstly, to the 
importance of grading pupils by using a variety of assessment methods, including 
individual and group grades and giving emphasis to individual over group performance 
and, secondly, to adequately debriefing projects by demonstrating reflection strategies 
and collecting formative evaluation information from pupils about the project and how it 
might be improved. 


Starting from the premise that project-based teaching assumes significant changes in classroom 
practices, Krajcik, Blumenfled, Mars and Soloway (1994) described how teachers can learn to 
address the new challenges presented through the dynamic interplay of three elements in middle 
school science teaching: teachers’ collaboration with consultants and university personnel to share 
and critique ideas, plans and teaching activities; classroom enactment where teachers plan and carry 
out new practices in the classroom in an attempt to construct and generate understandings about 
what is possible in their classroom, modify their thinking and adopt the most appropriate teaching 
strategies; teachers’ reflection on their teaching via journals, case reports or videotapes of classroom 
implementation to develop the knowledge that will help promote student learning. 
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Recommendations made on the basis of the evidence 


On the basis of the literature review, the following six key recommendations can be made which are 
considered to be essential for the successful adoption of a project-based learning approach in the 
mainstream school setting. 


1. Student support: pupils need to be effectively guided and supported; emphasis should be 
given on effective time management and student self-management including making safe 
and productive use of technological resources. 

2. Teacher support: regular support needs to be offered to teachers through regular 
networking and professional development opportunities. The support from the school 
senior management is crucial. 

3. Effective group work: high quality group work will help ensure that pupils share equal 
levels of agency and participation. 

4. Balance between didactic instruction with independent inquiry method work will ensure 
that pupils develop a certain level of knowledge and skills before being comfortably 
engaged in independent work. 

5. Assessment emphasis on reflection, self and peer evaluation: evidence of progress 
needs to be regularly monitored and recorded. 

6. Anelement of student choice and autonomy throughout the project-based learning 
process will help pupils develop a sense of ownership and control over their learning. 
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Appendix C: Consent and Information Sheet Examples 


Innovation Unit's Memorandum of Understanding with Control Schools 


Innovation 
Unit 


Memorandum of Understanding (MoU) between 
The Innovation Unit Limited and 


School (‘the Parties’) 


Parties 


° The Innovation Unit Limited (company registration number 05997039 and 
eqanass address of CAN Mezzanine, 49-51 East Road, London, N1 6AH) 
School, a school based at 


1. Purpose and Scope 


This MoU describes the framework within which and The 
Innovation Unit will work together on Learning through REAL Projects; 


Project description 

Innovation Unit has received a grant from the Education Endowment Foundation to 
run a Randomised Controlled Trial of Project Based Learning (which we call Learning 
through REAL Projects). This programme aims to raise attainment by engaging 
students more deeply in their learning. There is evidence of benefits of this approach 
from the US, but it has yet to be rigorously tested in the UK. A pilot of the support 
package is being run with 8 schools during the academic year 2013-14, 


The aim of this programme is to evaluate the impact of Learning through REAL 
Projects on students’ literacy and engagement outcomes. The results of the research 
will contribute to our understanding of what works in raising pupils’ attainment and 
will be widely disseminated to schools in England. Ultimately we hope that the 
evaluation will equip school staff with the skills to better design great learning to 
support and engage all students. 


The research design 
To find out what the impact is, Innovation Unit will compare a group of schools who 
implement Learning through REAL Projects against those that do not. 


24 Trial Schools have been randomly allocated into two groups of 12 (Group A and 
Group B) by evaluators at Durham University and the University of York. 


The process of randomly allocating schools into one of two groups is critical to the 


research design. Randomisation helps ensure that the two groups to be compared 
are as similar as possible, thereby reducing any possible bias in findings. 
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Selection to be one of the 24 Trial Schools was based on a number of factors 
including school capacity and commitment to high quality implementation of Learning 
through REAL Projects, From the perspective of the programme, schools in Group A 
and Group B are considered to be equal in every respect apart from the 
implementation (or not) of Learning through REAL Projects. 


Should any schoo! decide not to participate in the programme once randomization has 
occurred, it will not be possible to replace that school, and this could jeopardize the 
research. 


Group A schools will form a control group for the duration of the trial and Group B 
schools will prepare for and then implement Learning through REAL Projects from 
2014. 


2. Responsibilities 


as a school allocated to the Control Group (Group A), will 
be responsible for the following; 


¢ Not introducing Learning through REAL Projects practice with Innovation Unit 
support from September 2014 to July 2015, This is to ensure that there is 
sufficient distinction between the practice of the two groups of schools for 
meaningful conclusions to be drawn from the evaluation. 

« Allowing evaluators to conduct online or paper assessment tests with students at 
the end of academic year 2014/15, 

« Providing evaluators with additional information about the school and practice as 
requested, 


Use of Data 

A condition of funding by the EEF is that schools pass on their Unique Pupil Numbers 
(UPNs) to the evaluation team to allow the long term impact of the programme to be 
assessed. This is for all students in the Trial Schools taking part, from Group A and 
Group B. Parents and carers will however have the option to opt-out of their child’s 
data (including UPNs) being used for any part of the evaluation. 


Pupils’ test responses and any other pupil data will be treated with the strictest 
confidence. The responses will be collected by the evaluators, Durham University and 
the University of York. Named data will be matched with the National Pupil Database 
and shared with the EEF. No individual school or pupil will be identified in any report 
arising from the research. 


What as a school allocated to the Control Group (Group 
A) will receive from Innovation Unit in terms of support: 


e Designation as an EEF Research Partner School. 
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Regular updates on the progress of the trial. 

Data on student progress towards the end of Year 7 for all cohorts in the trial, 

provided by the independent evaluators, Durham University and the University of 

York. 

e Innovation Unit, REAL Projects and Education Endowment Foundation logos for 
inclusion on school websites and other promotional material. 

e There isno payment commitment for schools selected to Group A for the 
duration of the trial (see note on funding below). 

e Schools in Group A will additionally receive a stipend to cover the costs incurred 
as a result of submitting students for assessment purposes, not to exceed £1,000 
per school (see attached Purchase Order). 

e Innovation Unit will keep Control Group schools informed of international study 

visits and relevant programmes and events. 


3. Term 


° The start date of this agreement is 24 February, 2014. 
. The term of this MoU is for a period of 1 year and 5 months*, from the start 
date of this agreement until 31 July, 2015, 


*|mportant note on funding and the duration of the Trial 

Innovation Unit may secure funding for a second year of trial, and will contact Group 
A schools towards the end of academic year 2014/15 to let them know whether the 
trial will continue to academic year 2015/16. 


If it does, schools in Group A must continue to not introduce Learning through REAL 
Projects practise in academic year 2015/16. 


At this time, Group A schools may also decide to begin work with Innovation Unit to 
train staff and design projects from September 2015, in preparation for 


implementation from September 2016, This being the case, a new contract for 
services would be issued at this time. 


If the trial runs to two years, Group A schools also commit to allowing evaluators to 
conduct online or paper assessment tests with Year 7 and Year 8 students at the end 
of academic year 2015/16. 


Data on student progress for all cohorts in the trial, will be provided by the 
independent evaluators, Durham University and the University of York. 


4, Contracting and Financial Arrangements 
See attached Purchase Order. 
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5. Revisions 


Changes to this MoU can be made at any time if agreed by all the Parties and 
documented in writing. 


6. Indemnification 


Each Party agrees to defend, indemnify and hold harmless the other Parties, their 
officers, directors, employees, and agents from and against any and all claims, 
demands, actions, causes of action, penalties, judgments and liabilities (including court 
costs and reasonable legal fees) based upon or arising out of any act, omission, 
negligence, misconduct, or breach of any material condition of this agreement by the 
other party, its subcontractors, or their respective employees, representatives, 
servants, agents, invitees, licensees or contractors, 


7. Intellectual Property 


71 All Background Intellectual Property in Background Information used in 
connection with the Project shall remain the property of the party introducing the 
same (or their licensor) and such Background Information shall be clearly identified in 
writing to the other party. It should be noted for the purposes of this MoU that all 
materials, techniques and know how related to Project Based learning used by the 
project Based Learning Coaches are regarded as Background Intellectual Property. 


7.2 Each Party authorizes the other party to utilize relevant Background 
Intellectual Property, on a non-exclusive, royalty free basis, in order to develop the 
Foreground Intellectual Property. 


7.3 Under the terms of the Grant Agreement between Innovation Unit and The 
Education Endowment Foundation (EEF) — the main funders of this project - and the 
terms of the grant EEF received from the UK’s Department for Education, Intellectual 
Property Rights directly associated with grant funding from EEF are to be vested in 
the Crown. The Parties therefore irrevocably assign all Foreground Intellectual 
Property Rights by way of present assignment of present and future rights with full 
title guarantee to EEF. 


8. Disputes 
In the event of a dispute, the Parties shall attempt to work together in good faith to 


resolve such dispute, and failing that, each Party shall retain any rights they may have 
under existing, applicable law. 
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9. Confirmation of Agreement 
On behalf of 


Name: 


Signature: 


Position: Headteacher 


Date: 


On behalf of The Innovation Unit 


Name: Gareth Wynne 


Signature: 


Position: Company Secretary 


Date: 24 February, 2014 
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Innovation Unit’s Service Agreement with Intervention Schools 


SERVICES AGREEMENT 


between 


THE INNOVATION UNIT LIMITED 


and 


Education Endowment Foundation 


Project Based Learning 
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THIS AGREEMENT IS DATED 10 FEBRUARY 2014 
BETWEEN: 
The Parties 


(1) THE INNOVATION UNIT LIMITED a private company limited by 
guarantee, registered in England and Wales with company number 
05997039 and whose registered office is at CAN Mezzanine, 49-51 East 
Road, London N1 6AH (the “Innovation Unit”) and 

(2) a school based at 


This Agreement sets out the terms and conditions under which Innovation Unit will 
carry out the services as set out in Schedule 1 (“the Services”). 


1 Definitions 


1.1 In this Agreement the following expressions shall bear the following meanings: 


Background Information any information, know-how, methodologies 
or other materials introduced into, and used 
for the benefit of, the Project, by a party; 


Background Intellectual Ay Intellectual Property already in existence 
Property and held by either Party before this agreement 
commenced; 


means a day (other than a Saturday or a 
Sunday) on which commercial banks are open 
for general business in London; 


Business Day 


Effective Date means the date of this Agreement; 


Fees means the sums due on successful completion 
of each of the milestones set out in Schedule 
1 totalling £10,000 plus reasonable travel 
expenses plus VAT (at the prevailing rate); 
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FOI Legislation 


Foreground Intellectual 
Property 


Milestone 


Intellectual Property 


Project 


Request for Information 
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means the Freedom of Information Act 2000 
(FOIA), all regulations made under it and the 
Environmental Information Regulations 2004 
(EIR) and any amendment or re-enactment of 
any of them; and any guidance issued by the 
Information Commissioner, the Department 
for Constitutional Affairs, or the Department 
for Environment Food and Rural Affairs 
(including in each case its successors or 
assigns) in relation to such legislation; 


any Intellectual Property that arises or is 
obtained, licenced to or developed by either 
Party in connection with the Project, including 
without limitation any Intellectual Property ir 
the associated deliverables set out in Schedule 
1 and the Proposal; 


means the point at which the deliverables 
specified in Schedule 1 are to be completed; 


any patents, rights to inventions, copyright 
and related rights, trade marks, trade names 
and domain names, rights in designs, rights ir 
computer software, database rights, rights in 
confidential information (including know-how 
and trade secrets) and any other intellectual 
property rights, in each case whether 
registered or unregistered and all similar o1 
equivalent rights or forms of protection 
existing anywhere in the world which subsist 
or will subsist, now or in the future; 


means the project detailed in Schedule 1; 


shall have the meaning set out in the FOIA or 
EIR as relevant; and 
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Term the duration of the provision of the Services 
commencing on 10 February 2014 and ending 
on 31 July 2015. 


1.2 The headings of the clauses and schedules of this Agreement are used for 
convenience only and shall not affect the construction of this Agreement. The 
schedules form part of this Agreement. 

1.3 If there is an inconsistency between any of the provisions of this Agreement 
and the Proposal, the provisions of this Agreement shall prevail. 


2 Consideration 

2.1 In consideration of Innovation Unit performing the Services and subject to it 
performing its obligations under this Agreement and the provision of associated 
deliverables set out in Schedule 1, shall pay the Fees to Innovation Unit 
according to the payment schedule and timetable set out in Schedule 1. 


3 Innovation Unit’s Obligations 

3.1 Innovation Unit warrants and undertakes that it will, and will procure that any 

sub-contractor will: 
3.1.1 perform the Services during the Term in accordance with this 
Agreement and the Proposal and according to the timescales and conditions 
set out therein; 


3.1.2 at all times act in good faith towards and any other third 
parties; 
3.1.3 supply to , at such times as shall reasonably 


request, any information howsoever relating to the Services that 

may require from time to time (“Information Request”); 
3.1.4 for each Information Request, provide such further information as is 
reasonably requested by 
3.1.5 perform the Services according to the highest professional standards 
expected of a competent supplier of services identical or similar to the 
Services and in a diligent, competent and co-operative manner and ensure 
that any reports provided are of high quality. 
3.1.6 keep informed as to the progress of the Services, or any 
problems experienced and to inform promptly of any new 
developments or of any potential opportunities that Innovation Unit may 
identify from time to time; 
3.1.7 on Customer’s request, return any equipment given by Customer to 
Innovation Unit and treat such equipment with reasonable care and only use 
such equipment for business purposes relating to the provision of the 
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Services; 

3.1.8 perform the Services in accordance with all applicable laws, rules and 

regulations; 

3.1.9 perform the Services in an ethical and trustworthy manner in all 

respects and in particular (but without limitation), Innovation Unit will: 
3.1.9.1 behave honestly and courteously to any persons Innovation 
Unit may come into contact with whilst performing the Services; 


3.1.9.2 update on a regular basis, including being honest and 
open about any difficulties that it may have encountered; 

3.1.9.3. provide any or all information required by for the 
purposes of effective monitoring of the progress and quality 
of the Services being provided; 

3.1.9.4 meet with or its nominated representatives at such 
reasonable intervals as shall request, to discuss the 


performance of the Services and their progress; 
3.1.9.5 nominate Louise Thomas(the “Project Director”) to oversee 
the provision of the Services and act as 's key contact. The 
Project Directors shall have all necessary expertise and have authority to 
make decisions on Innovation Unit's behalf with regard to the Services. 
Innovation Unit agrees that in the event that Louise Thomas cannot for 
any reason continue as Project Director it will agree a suitable 
replacement with : 
3.1.9.6 ensure that the persons appointed to provide the Services are 
suitably qualified and are able to dedicate sufficient time and attention to 
allow the Project to progress in line with the timescales set out in the 
Schedule 1 and the Proposal; 
3.1.9.7. allow such staff from or any agents of to 
contribute to, assist with or participate in the provision of the Services as 
deems at its sole discretion necessary. 


4 ‘s Obligations 

4.1 shall give clear and timely instructions and information to 
Innovation Unit to enable Innovation Unit to provide the Services. 

4.2 agrees to implement learning through real projects as per Schedule 
i. 

4.3 will be responsible for ensuring participating staff are released for 


appropriate time on the agreed days including the training session for teachers. 


5 Fees 

5.1 All sums payable to Innovation Unit under this Agreement are exclusive of VAT 
which shall, where applicable, be payable on submission of an appropriate VAT 
invoice at the prevailing rate. 

5.2 The Fees shall be inclusive of all expenses incurred by Innovation Unit in 
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connection with the Services, and Innovation Unit shall be responsible for all costs 
and expenses incurred in this respect. 


5.3 will settle invoices approved by it within 30 days of receipt, 
provided always that such invoices are submitted in accordance with the 
Milestones. 

5.4 No refunds will be offered in the event that chooses to withdraw 
from being a Implementation School and the full fees will still be due on milestone 
dates. 


6 Warranties 

6.1 Innovation Unit hereby warrants and agrees that: 
6.1.1 it has or will obtain all rights, licenses, consents, approvals, 
permissions, permits, tests and other certificates and authorities (whether 
public or private) necessary to perform the Services; 
6.1.2 any materials produced in performing the Services or developing the 
Project will not infringe the intellectual property rights (or any other rights) of 
any third party; 
6.1.3 it has not licensed or assigned any rights in the Background Intellectual 
Property or the Foreground Intellectual Property to any third party in any 
part of the world; 
6.1.4 it has the appropriate time, skills, experience and resources to perform 
and deliver the Services in accordance with this Agreement; 
6.1.5 all information, data and materials provided at any time by or on its 
behalf to is true and accurate; 
6.1.6 it has full power and authority under its constitution and has taken all 
necessary actions to execute and perform this Agreement; 
6.1.7 it and any sub-contractors engaged in accordance with clause 14.1 do 
not have any actual or potential conflict of interest with (including 
its trustees, officers or employees) in entering into and carrying out its 
obligations in accordance with the terms of this Agreement and Innovation 


Unit will immediately inform if Innovation Unit becomes aware of 
any actual or potential conflict of interest, whereupon it will be at “s 
discretion whether or not to continue the arrangements as set out in this 
Agreement. 

7 ~~ Publicity 

7.1 Innovation Unit will comply with ’s publicity requirements from time 

to time. 

7.2 will comply with EEF’s publicity requirements from time to time. 


8 Duration 
8.1 The terms of this Agreement shall become effective on the Effective Date and, 
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subject to this clause 8, shall continue in effect for the duration of the Services. 

8.2 Either party may terminate this Agreement immediately by giving the other 
notice in writing if the other has committed a material or persistent breach of its 
obligations or ceases for any reason to carry on business, or is unable to pay its 
debts as they fall due, or has a receiver, manager, administrator administrative 
receiver or liquidator appointed in relation to its assets. 

8.3 Termination of this Agreement shall not prejudice the accrued rights and 
obligations of either party. The following paragraphs shall continue in force following 
termination: 6-22 inclusive. 


9 Indemnity and Limitation of Liability 

9.1 ’s total liability to Innovation Unit, whether in contract, tort 
(including negligence) or otherwise in connection with this Agreement shall in no 
circumstances exceed a sum equal to the Fees paid or payable, provided always 
that nothing in this Agreement shall limit the liability of either party for death or 
personal injury caused by its negligence, for fraudulent misrepresentation, or for 
breach of any of the obligations under the Sale of Goods Act 1979 or the Supply of 
Goods and Services Act 1982 which may not be excluded by contract. 


10 Rights of Third Parties 
10.1 A person who is not a party to this Agreement may not rely upon it or 
enforce any rights pursuant to the Contract (Rights of Third Parties) Act 1999, 


11. Provision of Information and Freedom of Information 


11.1 In the event that is subject to the provisions of Freedom of 
Information Act, Innovation Unit will, on written request from , provide 
with any information relating to the Services as may 


reasonably require. 


12 Intellectual Property 

12.1 All Background Intellectual Property in Background Information used in 
connection with the Project shall remain the property of the party introducing the 
same (or their licensor) and such Background Information shall be clearly identified 
in writing to the other party. It should be noted for the purposes of this contract 
that all materials, techniques and know how related to Project Based learning used 
by the project Based Learning Coaches are regarded as Background Intellectual 
Property. 

12.2 Each Party authorizes the other party to utilize relevant Background 
Intellectual Property, on a non-exclusive, royalty free basis, in order to develop the 
Foreground Intellectual Property. 

12.3 Under the terms of the Grant Agreement between Innovation Unit and The 
Education Endowment Foundation (EEF) — the main funders of this project - and 
the terms of the grant EEF received from the UK’s Department for Education, 
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Intellectual Property Rights directly associated with grant funding from EEF are to be 
vested in the Crown. The Parties therefore irrevocably assign all Foreground 
Intellectual Property Rights by way of present assignment of present and future 
rights with full title guarantee to EEF. 


13 Confidentiality 
13.1 Innovation Unit agrees at all times to keep strictly secret and confidential all 
information and materials (in whatever form) disclosed to Innovation Unit by or on 
behalf of (“Confidential Information”), and not at any time for any 
reason whatsoever (whether before or after termination or expiry of this 
Agreement) disclose the Confidential Information to any third party and only use it 
for the purposes of performing the Services. 
13.2 Innovation Unit shall procure that all of its employees, agents, 
sub-contractors and professional advisors having access to any of the Confidential 
Information shall be subject to the same confidentiality obligations as set out in this 
clause 13. 
13.3 The obligations of confidence referred to in this Agreement shall not apply to 
any information which: 

13.3.1is required to be disclosed by law; or 

13.3.2 is already lawfully in the public domain; or 

13.3.3 Innovation Unit can prove by documentary evidence was already in its 

possession; or 

13.3.4 is disclosed by Innovation Unit with the express written consent of 


14 Assignment and Variation 
14.1 Innovation Unit may not assign or sub-contract any of its obligations set out 


in this Agreement, without 's prior written consent. 
14.2 Any amendment to the terms of this Agreement must be agreed in writing by 
both parties. 


15 Disrepute 
15.1 Innovation Unit will not, either by act of neglect or default, do anything which 
is or may reasonably be expected to be injurious to the interests or reputation of 


16 Legal relationships 

16.1 Nothing in this Agreement creates a partnership, a joint venture, the 
relationship of employer and employee, or establishes a relationship of principal and 
agent or other fiduciary relationship between the parties to this Agreement, and 
neither nor Innovation Unit will hold itself out as having any authority to 
incur any obligation of any nature whether express or implied on the other's behalf. 


17 Further Assurance 
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17.1 Innovation Unit shall support to obtain the full benefit of this 
Agreement, whether in connection with any registration of title or other similar right 
or otherwise. 


18 Prevention of Corruption 

18.1 Innovation Unit shall at all times and shall procure that any sub-contractors 

engaged in accordance with clause 14.1 shall at all times: 
18.1.1 act lawfully, ethically and in good faith in its dealings with 
and comply fully with the Prevention of Corruption Acts 1889 to 1916, the 
Bribery Act 2010 and any related regulations rules, procedures or like 
legislation; and 
18.1.2 report promptly to any fraud, theft and/or corruption of 
which Innovation Unit becomes aware or suspect to have occurred in relation 
to the provision of the Services. 

18.2 Innovation Unit shall ensure that, at any time when its employees or 


sub-contractors are on 's premises, they comply in full with 's 
policies and procedures relating to health, safety and security and any instructions 
given to Innovation Unit by or its officers in that regard. 


19 Entire Agreement 

19.1 This Agreement, constitutes the whole agreement between ourselves relating 
to its subject matter and supersedes and extinguishes any prior drafts, 
agreements, undertakings, representations, warranties and arrangements of any 
nature, whether in writing or oral, relating to such subject matter provided that this 
paragraph 19.1 shall not operate to exclude either party's liability to the other for 
fraudulent misrepresentation. In the event of any inconsistency between the terms 
of this Agreement and the Proposal, the terms of this Agreement shall prevail. 


20 Waiver 

20.1 Failure or neglect by either party to enforce at any time any of the provisions 
of this Agreement shall not be construed nor shall it be deemed to be a waiver of 
our respective rights hereunder, nor in any way affect the validity of the whole or 
any part of this Agreement, nor prejudice our respective rights to take subsequent 
action. 


21 Force Majeure 

21.1 Innovation Unit shall not be liable to in respect of anything which, 
apart from this provision, may constitute a breach of this Agreement arising by 
reason of circumstances outside of Innovation Unit's reasonable control (“Force 
Majeure”). 

21.2 Innovation Unit shall give to notice as soon as practicable upon 
becoming aware of an event of Force Majeure which prevents performance of its 
respective obligations under this Agreement, and of the cessation of the said event. 
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21.3 If the Force Majeure event prevails for a continuous period of more than one 
month, may terminate this Agreement by giving 14 days’ written notice 
to Innovation Unit. On the expiry of this notice period, this Agreement will 
terminate. Such termination shall be without prejudice to the rights of the parties in 
respect of any breach of this Agreement occurring prior to such termination. 


22 Governing Law and Jurisdiction 

22.1 Both parties agree that this Agreement shall be governed by and interpreted 
in accordance with English Law, and hereby submit to the exclusive jurisdiction of 
the English Courts. 


Signed by Gareth Wynne 


Company Secretary : | 
For and on behalf of Innovation C( 
Unit 

y 
Dated 10 February 2014 
Signed by 


CAMO RICUIATE CHE” =i senpnascenassonsonnassensnscensasibscbensctnndapenansenssszonte 


Dated: 
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Schedule 1 


The Project 


Learning through REAL Projects 
Trial School Contract Schedule 
for schools allocated to Implementation Group (Group B) 


Project description 


Innovation Unit has received a grant from the Education Endowment Foundation to run a 
Randomised Controlled Trial of Project Based Learning (which we call Learning through REAL 
Projects). This programme aims to raise attainment by engaging students more deeply in their 
learning. There is evidence of benefits of this approach from the US, but it has yet to be 
rigorously tested in the UK. A pilot of the support package is being run with 8 schools during the 
academic year 2013-14. 


The aim of this programme is to evaluate the impact of Leaming through REAL Projects on 
students’ literacy and engagement outcomes. The results of the research will contribute to our 
understanding of what works in raising pupils’ attainment and will be widely disseminated to 
schools in England. Ultimately we hope that the evaluation will equip school staff with the skills 
to better design great learning to support and engage all students. 


The research design 


To find out what the impact is, we will compare a group of schools who implement Learming 
through REAL Projects against those that do not. 


24 Trial Schools have been randomly allocated into two groups of 12 (Group A and Group B) by 
evaluators at Durham University and the University of York. 


The process of randomly allocating schools into one of two groups is critical to the research 
design. Randomisation helps ensure that the two groups to be compared are as similar as 
possible, thereby reducing any possible bias in findings. 


Selection to be one of the 24 Trial Schools was based on a number of factors including school 
capacity and commitment to high quality implementation of Learning through REAL Projects. 
From the perspective of the programme, schools in Group A and Group B are considered to be 
equal in every respect apart from the implementation (or not) of Learning through REAL 
Projects. 

Should any school decide not to participate in the programme once randomization has occurred, 
it will not be possible to replace that school, and this could jeopardize the research. 
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Group B will prepare for and then implement Leaming through REAL Projects from 2014, with 
Group A forming a control group. 


What schools allocated to the Implementation Group (Group B) will receive from 
Innovation Unit in terms of support 


All 24 Trial Schools will receive: 
e Designation as an EEF Research Partner School 
e Regular updates on the progress of the trial 
e Data on student progress towards the end of Year 7 for all cohorts in the trial, provided by 
the independent evaluators, Durham University and the University of York 
e Innovation Unit, REAL Projects and Education Endowment Foundation logos for inclusion 
on school websites and other promotional material. 


In addition, schools allocated to Group B will receive support from Innovation Unit to implement 
Learning through REAL Projects as follows: 


Preparation support package (Jan-Jul 2014): 

e 3 day residential induction training on Learning through REAL Projects for up to 8 
members of staff. 

e One day whole staff INSET (subject to availability of coaches). 

e 3days leadership support from expert coaches and former headteachers. 

e 3days Leaming through REAL Projects support from expert coaches from High Tech 
High. 

° — to meet other schools in Group B through events and online collaboration. 

e Visits to Innovation Unit Developer Schools already implementing Learning through 
REAL Projects. 


sash support package (from Sept 2014): 
18 days support by High Tech High coaches in academic year 2014/15 (one day per 
school per fortnight); (12 days in academic year 2015/16"). 
e Up to 9 days of leadership support (3 per term) in academic year 2014/15; (4 days in 
academic year 2015/16"). 
e Access to online and face-to-face opportunities to network with other schools. 
Tools and resources. 


What schools allocated to the Implementation Group (Group B) contribute to the costs of 
the programme 


Schools randomized to Group B commit to: 


e A period of preparation and a minimum of 1* year of implementation at a cost of £10,000 
+ VAT plus reasonable travel expenses incurred by coaches. 


e Covering staff travel and accommodation expenses and any supply cover to enable staff 
participation in the programme. 
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What schools allocated to the Implementation Group (Group B) commit to delivering as 
part of the programme 


Schools randomized to Group B commit to: 


Delivering Learning through REAL Projects to Year 7 cohort starting in September 2014 
(moving to incoming Y7 and Y8 from 2015*) (see ‘Definition’ of REAL Project Based 
Leaming outlined in the Learning through REAL Projects - Handbook for Trial Schools). 


Implementing changes to staffing and timetable structures according to the ‘Minimum 
Implementation Commitments’ detailed in the Learning through REAL Projects - 
Handbook for Tnal Schools. 


Releasing all relevant staff for at least 6 days training during the spring and summer 
terms 2014 (3 days formal induction training plus 3 days school based project design 
work). 


Allowing coaches from High Tech High and Innovation Unit to work with leaders, 
teachers and students to develop outstanding REAL Project Based Learning practice. 


Allowing evaluators to conduct online and paper assessment tests with students at the 
end of academic year 2014/15 (and academic year 2015/16"). 


Providing evaluators with additional information about the school and practice as 
requested. 


“Important note on funding and the duration of the Trial 


Innovation Unit may secure funding for a second year of trial, and will contact Group B schools 
towards the end of academic year 2014/15 to let them know whether the trial will continue to 
academic year 2015/16. 


If it does, schools in Group B will continue to receive the support above in academic year 
2015/16. 


If the trial runs to two years, Group B schools also commit to allowing evaluators to conduct 
online or paper assessment tests with Year 7 and Year 8 students at the end of academic year 
2015/16. 


Data on student progress for all cohorts in the trial, will be provided by the independent 
evaluators, Durham University and the University of York. 
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Use of Data 


A condition of funding by the EEF is that schools pass on their Unique Pupil Numbers (UPNs) to 
the evaluation team to allow the long term impact of the programme to be assessed. This is for 
all students in the Trial Schools taking part, from Group A and Group B. Parents and carers will 
however have the option to opt-out of their child's data (including UPNs) being used for any part 
of the evaluation. 


Pupils’ test responses and any other pupil data will be treated with the strictest confidence. The 
responses will be collected by the evaluators, Durham University and the University of York. 
Named data will be matched with the National Pupil Database and shared with the EEF. No 
individual school or pupil will be identified in any report arising from the research. 


Milestones and invoicing schedule 


The Fees will be payable upon delivery of the deliverables outlined above by the 
Milestone dates set out below and in accordance with the terms of this Agreement. 


On signature of contract by 
customer. 


£5,000 + On completion of services 1 July 2015 
travel 

expenses + 

VAT 
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School information sheet, recruitment 


Innovation Learning through A 
Unit REAL PROJECTS 
Dear [name of headteacher] 


Innovation Unit is seeking 24 schools eager to transform theirteaching and learning in Key 
Stage 3. 


This is your opportunity to participate in a national programme that raises achievement by 
increasing student engagement in learning. 


Weare very pleased to invite [name of school/alliance / cluster /trust] to register your interest in 
being one of those 24 schools by 31 July 2023. 


We have received a grant from the Education Endowment Foundation to run the first major trial of 
Learning through REAL Projects inthe UK. This is one of the larger and more ambitious projects 
supported by the Foundation. 


Learning through REAL Projects uses student enquiry, feedback and public exhibition of student 
work to promote deep learning of subject knowledge and learning that has a real impact on the 
world outside school. Please see the enclosed summary for more information on REAL Projects. 


There is evidence from the US that Learning through REAL Projects has a strong positive impact on 
student achievement, student engagement in learning and students’ success at College and 
University. Please see the enclosed summary for more information about our US partners High 
Tech High, world leaders in educational innovation. 


Innovation Unit currently employs teachers from High Tech High to support pioneering schools in 
England to develop world class practice. We are now searching for 24 new schools to participate in 
this trial. Each trial school will receive: 


training for classroom teachers in Learning through REAL Projects, 
in-school coaching from High Tech High teachers, 

leadership coaching and support from Innovation Unit Senior Associates, 
data, research and evaluation from University of Durham, and 

an opportunity to learn from and develop your practice with other schools. 


There are two phases to the trial. If selected to participate, your school will be randomly allocated to 
one of the two phases: 


Phase a: Begins January 2024, withimplementation with Year7 from September 2014 (next year) 
Phase 2: Begins September 2025, with implementation with Year7 from September 2026. 


Although the summer break is so close, it would really help us to plan forthe autumn term and to 
guarantee your place on our register, if you could return your statement of interest by 31 July 
2013. This only requires minimal information from you at this stage. All interested schools will be 
invited to an event in October 2023 to find out more, and schools will be chosen and notified by 
January 2014. Please respond using the enclosed Register of Interest form. 


We really look forwardto hearing from you. 
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Parent opt-out letter 


Rovean House 

Mountjoy Centre 

Durham Uni iversty, Stockton R Road 
Durham woe 

+44 (0) 191 9244682 
WWW.CenLore 

23rd January 2015 


Dear Parent/Carer 
Learning through REAL Projects Evaluation by Durham University 


| am writing to let you know about a research study which will be taking place in your child's school during the 
2014/15 school year. The school has signed up to be involved with the ‘Learning through REAL Projects’ initiative 


through 
based learning this year. The school will be able to provide more information on what this means for your child. 


Durham University are funded by the Education Endowment Foundation (EEF) to evaluste the impact of the 
“Learning through REAL Projects’ initiative on pupils” learning and their attitudes to school. The head teacher of the 
school has given permission forthe schod to take part, and the study willinvolve all pupils in Year 7. 


Wa seal ast paris to comynete a asaoy as cesar at tie ed of the yout vane 2810. The assessments 
will be marked by GL Assessments and the results will be given to the researchers st D. ry mlirsomerig AIA Lina 
forward the results to the school to informteaching. We will also ask pupils to complete 3 
Sttitudes to school and learning. We plan to match the dats from the assessments to the "ray Sige aaa 
and their future achievement through linking our results with the Nations! Pupil Database 


To help us with this research the school will pass DA SO: Recnraae ace ne es Puen date of 


i on ( 
Free School Meal status) fromthe NPD. All data collected willbe treated confidentally and will only be used for 
research purposes. The dats for your child's school will be analysed anonymously, together with data from other 
schools, and no individual pupils or schools will be named in any report. We will share data with our partner 
researchers at the University of York, << 
data managers and the UK Data Archive for future research 


| do hope that we have your support for ths important study and that you are happy for your child's data to be used 


in this way. If you would prefer we did not use your child's data in this research study please contact the school to 
opt out at any time during the project. You can also contact the research team directly using the contact details 
below. 


If you have any questions about the study please contact Clare Collyer, Project Administrator on 0191 334 4682, or 
email clare. collyerggcem.dur.ac. uk. 


Yours faithfully, 


An 
WP Durham 


University 
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Pupil focus group, Parent opt-in 


CEM 
t [ } 


A 
‘ 4 


Durham, UK DH! 3UZ 
Tel: +44 (0) 191 3344682 


www.cemorg 


iLearning through REAL Projects Evaluation by Durham University 
Dear Parent'Carer 


We are writmg to the parents/carers of pupils m Year 7 m your child’s school. Durham University has 
been funded by the Education Endowment Foundation to evaluate the Leammg through REAL Projects 
programme which <school> is taking part m this year. We want to find out more about what pupils think 
of leammg im this way at school and to look at any improvements that could be made to the programme m 
the future. As part of our evaluation, researchers from the university will be visitmmg the school over the 
course of the next few months. This will be to see how teaching through projects works m practice as 
well as to talk to some of the pupils and staff mvolved. 


We would like to hold a group discussion (or focus group) with some Year 7 pupils. This will mvolve two 
researchers from the university meetmg with a small group of pupils (5-6 children) accompanied by 2 
member of staff from the school. We will ask the children questions about their experiences of domg 
different projects at school, mcluding tellmg us what they enjoyed about each project and what they 
leamed. We expect the focus group to last for about half an hour and to take place during school tme. 
The discussion will be recorded so that we can write up what is said afterwards; however, the recordmg 
will be deleted after notes have been taken from it. All comments and responses will be treated 
confidentially and no mdividuals will be named m any report about the project. Participation m the focus 
group is voluntary and pupils will be given the choice to take part on the day. 


If you are happy for your child to be mvolved m a focus group please complete the tear off sheet below 
and retum it to the school, who will pass it on to the research team. 


If you need further mformation about the evaluation or have any questions please contact the Research 
Team through Clare Collyer, Project Admmistrator: clare.collyer@cem.dur.acuk: 0191 334 4682. 


Yours faithfully 


I give consent for my child to be mcluded m a focus group for the Leaming through REAL 
Projects evaluation to find out about the pupils® experiences of leammg m this way. 
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Appendix D: Attitudes to School and Learning 
Questionnaire 


Attitudes to School and Learning Questionnaire 
Independent Evaluation of Learning through REAL Projects 
Programme 


We would like to learn about how you feel about school and your attitudes to learning 
generally. This information is being collected as part of a research project being done at 
Durham University which is evaluating different ways of teaching and learning in schools. 
We are working with twenty four different schools and looking at whether the way some 
schools teach different subjects can changes pupils’ attitudes. 


The answers you give will be kept confidential. We will collect the questionnaire in today 
and it will be sent back to the university. We won’t share your individual answers with your 
teachers. We will keep your information secure. 


The questions require you to indicate how true the statement is for you. Try to be as honest 
as you can. There are no right or wrong answers. 


If you don’t want to answer a question just leave it blank and move onto the next one. 
Please ask any questions you have by putting your hand up. 


You will have up to half an hour to complete the questionnaire. 


About you 


First Name 


Surname 


Date of birth 


School 


Gender Male Female 


(please tick) 
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About school 
Very true Mostly true 


1. | feel like a part of the school. 


2. Other pupils in this school take my opinions seriously. 


3. There's at least one adult in the school | can talk to if | 
have a problem. 


4. | can really be myself at school. 
5. My teachers know | can do good work. 
6. Most mornings | am keen to go to school. 


7. School is a nice place to be. 


8. All pupils at this school have a good chance to be 
successful in the future. 


9. | respect my teachers. 

10. | make friends easily at school. 

11. At school teachers treat pupils fairly. 

12. Teachers at this school care about the pupils. 
13. My teachers are open and honest. 

14. Teachers listen to pupils' ideas. 

15. Pupils help each other learn. 


16. Pupils are friendly to each other. 


Education Endowment Foundation 


Not very 
true 


Not at all 
true 
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Motivation at school 


Mostly 
true 


Very true 
17. | like work that really makes me think. 
18. | enjoy learning new things. 
19. | try to learn from my mistakes. 


20. | always try to get the best marks possible. 


21. What we learn at school makes me interested to 
learn about new things. 


22. | normally enjoy my classes. 
23. | am interested in what | learn at school. 


24. What | learn at school will be important in my 
future. 


25. My school work is useful to me. 


How | do at school 


Mostly 
Very true mm 
26. | think | will get good marks in school. 


27. | find it easy to understand what is taught at 
school. 


28. | am confident in my ability to succeed in school. 
29. | think I'll get good marks in maths. 
30. | think I'll get good marks in science. 


31. | think I'll get good marks in English. 
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Not very Not at all 
true true 


Not very Not atall 
true true 
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Getting involved in school 


Very true 


32. | talk with people outside of school about what I'm 
learning in class. 


33. In class, | work as hard as | can. 

34. | get involved in class discussions. 

35. | pay attention in class. 

36. | often answer questions in class. 

37. | take pride in the quality of my school work. 
38. I'm happy to ask my teachers for help. 

39. In class, | ask questions to help me learn. 


40. | let my teachers know what I'm interested in. 


41. Sometimes | get so interested in my work | don't 
want to stop. 


42. | help to choose what we learn about. 


43.1 am enthusiastic about most of the things we do 
in class. 


44. Sometimes | do extra work outside of school 
because I'm interested in the topic. 


45. | often find myself thinking about what we are 
learning after a lesson is over. 


46. | carry on learning when I'm not at school. 
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Mostly 
true 


Not very Not at all 
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My school work 


47. After finishing a piece of work | check it carefully 
to make sure it's as good as possible. 


48. | try to understand how different ideas fit together. 
49. | plan my work before | start. 


50. | try to do well on my schoolwork even when it 
isn't interesting to me. 


51. When I'm doing work for school | try to connect 
what I'm learning with my own experiences. 


52. | always make sure that | understand the question 
or task before | start work. 


53. | ask other people for ideas about how to improve 
my work. 


Education Endowment Foundation 


Very true 


Project Based Learning 


Mostly 
true 


Not very Not at all 


true 


true 
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Appendix E: Case studies in two schools doing PBL 


Case studies 
Introduction 


PBL is a complex intervention which requires commitment and effective working at many levels in 
participating schools. This section seeks to provide an insight into how the various essential elements 
of PBL can come together to provide effective implementation. We look at two schools where it was 
agreed by both the Innovation Unit, as developer and delivery partner, and the evaluation team, that 
the implementation of REAL Projects has been effective. Both schools also agreed that their 
implementation has been effective. Importantly whilst there are quite substantial similarities between 
the schools, the implementation has nonetheless been different in a number of respects — supporting 
the notion that there is no simple ‘one size fits all’ version or implementation of PBL. 


Background to the schools 


Both schools are larger than average, well established and draw the majority of their pupils from 
nearby communities on the outskirts of large conurbations. There were relatively small differences in 
their last Ofsted ratings (2 and 3 with one having gone up, and the other down) but nonetheless both 
were judged to be reasonably effective with good leadership. In both instances, English teaching at 
KS3 was identified as an area that could be improved, which is the primary outcome for the trial. 
Attitudes to learning and support from parents were judged to be good, as was attendance and 
relationships between pupils and teachers. It can therefore be reasonably assumed that from an 
Ofsted perspective both schools were in a good position to implement and manage the necessary 
organisational changes required for PBL. 


School E 


The starting point for school E was an ongoing internal review of their curriculum and wider school 
development, and an identified need to improve KS3 teaching. Some elements of ‘project work’ and 
student-led or independent learning were already being undertaken and the school as a whole was 
keen to encourage similar initiatives. In such circumstances it was not surprising that the offer to be 
included in the trial, in particular the intervention arm, was enthusiastically accepted. The headteacher 
emphasised that the school was looking for something like PBL to provide more structure and rigour 
to their plans and aspirations for KS3. 


Organisation and challenges 


Both the headteacher and the PBL lead, as well as several of the teachers we spoke to, reported that 
one of the biggest challenges of doing the PBL project was the very tight timescale. In practice this 
amounted to signing up in February with training taking place in May/June and starting to deliver 
REAL Projects the following September. This was necessary for the timing of the trial, but there was a 
consensus that at least a term or two is needed to plan for the necessary organisational changes, and 
ideally schools should be looking a year ahead when? planning to implement REAL Projects. It follows 
that quite a few schools would not have been able to undertake the necessary preparatory work in this 
short time period, and that this school was able to is a reflection on the school, and should be 
considered when generalising. The school felt in a secure position internally and externally to address 
the challenges of a major new initiative such as PBL. 


A major perceived challenge identified early in the planning phase was getting the necessary ‘buy-in’ 
and support from three departments not used to working together. They found that the departments 
were able to work well together but they emphasised that it was important to ensure that all three were 
equally valued and time taken for REAL Projects from subject teaching accordingly. Once the project 


Education Endowment Foundation 96 


Project Based Learning 


was underway, most aspects of concern centred on logistical issues. Whilst it was felt that some of 
these could have been better planned for, in practice it seemed likely that they would have occurred 
with any major organisational change. It was also emphasised that many staff needed additional 
support when working outside of their ‘comfort zone’, which included working in a cross-curricular way 
as well as with new subjects and teaching such as ICT. For the Year 7 pupils, secondary school 
generally was a new experience, however it was emphasised that their “student voice” survey 
indicated pupils very much enjoyed working in a project way (comparison was made to the previous 
year as well as other non-PBL subjects). 


An important commitment was made at an early stage of planning, that the curriculum needed to drive 
the timetable, even when this might have knock on effects on other subjects. In practice the REAL 
Projects curriculum was not found to constrain wider school working significantly. The pupils did PBL 
twice a week for 150 mins at a time (i.e. triple 50 min lessons). One session was humanities focused 
and the other Business and ICT. This was found to be a good and effective arrangement which fitted 
well with the wider school organisation and will continue for future years. 


The staff we spoke to emphasised the value and importance of headteacher, senior staff and 
governor buy-in to the pedagogy. Again, this largely reflects the nature and values of the school and 
importantly was identified by everyone we spoke to at the school as an essential pre-requisite for 
effective PBL. There was quite considerable emphasis on the need for planning and meeting time at 
the school, in particular when there were up to 12 teachers delivering PBL. The school also placed 
considerable importance on the role of teaching assistants (TAs) to support the pupils generally as 
well as individually for those with specific needs. Indeed we saw one lesson where the pupils had 
reading ages ranging from 6 to 15 years as well as EAL pupils, and in such instances TAs played a 
very important role. 


Training and Innovation Unit support 


The school was impressed with and highly valued the support from the Innovation Unit, both in terms 
of the Leadership and Classroom Coaches, although in different ways. The Classroom Coaches were 
found to be very approachable and knowledgeable, and importantly were happy to work alongside the 
teachers. The Leadership Coach, needed to adopt a role ‘sympathetic’ to the needs of the 
headteacher and leadership team, and this school felt it worked well. 


In addition to two of the staff attending the QUEST training in London, the Classroom Coach 
replicated the programme in the school for most of the REAL Projects teaching team. This took place 
over a two day period and was found to be highly productive and beneficial, and importantly helped 
bring the staff together for PBL in a way that just one or two teachers reporting back to the rest of the 
staff would not have done. 


It was felt that the Classroom Coach had unique skills and experience that was unlikely to be available 
currently in U.K. based trainers. However, the Leadership Coach skills were viewed as more generic, 
and could therefore be ‘bought in’ locally or regionally. Nonetheless it was suggested that some of the 
school’s PBL staff would be able, in time, to share their PBL skills and expertise with other schools. 


Perceived benefits and pupil progress 


The school largely followed the Innovation Unit specifications and suggested projects in the first year 
of delivery, but started to vary their offering and approach in the second year. This was felt to be of 
more value to the staff and the school’s development more generally as it was more targeted at the 
school’s needs. This also helped departments to think and work together and therefore had positive 
benefits in other subjects and areas of the school, potentially leading to benefits pupils beyond Year 7 
and 8. 
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It was also found that REAL Projects supported the existing primary /secondary transition activities, as 
well as wider local contacts and educational opportunities. For instance, the school had worked with a 
local museum and local designers in some of the projects. Likewise, a number of staff visited other 
PBL schools, both locally and further afield, which they suggested helped bring in new ideas. More 
generally, working in a team was found to help many staff reflect on their own practice and skills and it 
was thought it would therefore support staff development more generally. For the pupils’ progress 
through the school it was felt that the PBL approach to learning instilled valuable learning skills, such 
as research skills, as well as giving them an introduction to subject / department organised teaching, 
which they would not have been familiar with from primary school. 


As with many of the other schools there were some challenges in assessing impact. They found that 
their approach in the first year of delivery lacked consistency and it was difficult to draw firm 
conclusions about progress. However, in the second year, the systems were more aligned and 
integrated with the whole school approach which allowed them to make direct comparisons both year 
on year and between subjects. The school emphasised that consistency in assessment and the ability 
to compare progress was important for maximising the returns from PBL, as well as giving wider 
credibility to the approach. 


In general, feedback from the various departments suggested that the Year 7 group as a whole had 
made more progress than previous years. It was also felt that PBL encouraged deeper leaning of the 
particular topics and increased the breadth of content that could be explored by going beyond the 
national curriculum. Specifically it was felt PBL helped the development of a wide range of learning 
skills which would be useful throughout the pupils’ education, such as research, team work and 
presentational skills. The school had also found the PBL approach more suited to low-ability learners 
and were able to point to instances of far more progress being made by such learning than would 
have been expected. However, they did recognise this was not in itself a solution for all such pupils 
and more planning and adaption of lessons needed to be done. 


Conclusion 


This school was very enthusiastic about PBL and felt that it was of substantial benefit to their teaching 
and learning. They were also positive about the potential benefits to other schools, however they did 
point to the need for the school to be in a ‘secure’ position, both internally and externally. From their 
experience, a great deal of commitment had to be made to properly implement the project, and to 
reap the benefits for pupils. 


School F 


This school was motivated to do REAL Projects in order to holistically improve educational outcomes 
for pupils leaving secondary school, both in terms of exam outcomes but also promoting skills 
required for employment and further / higher education (including in particular STEM subjects) as well 
as local needs. They felt the architecture for what they wanted was largely already in place, but that 
the curriculum, both nationally and how it was being interpreted locally, was not able to support the 
outcomes they desired. The school had a long term interest in initiatives such as the CBI 21° Century 
Skills and the more recent RSA Opening Minds competencies framework, which mesh well with the 
underlying philosophy and principles of PBL. As the head and PBL lead emphasised, pupils come 
from primary schools with a passion to learn and it is essential they put in place a way for that passion 
to continue and thrive in secondary school. 


Organisation and challenges 


In addition to having much of the organisational structure in place for PBL, the PBL lead had recently 
completed a Masters degree, focusing on how STEM could be effectively taught at the school through 
PBL, when they were approached by the Innovation Unit to join the trial. This, combined with support 
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from the governors, made it relatively straightforward (compared to other schools) to implement PBL; 
however, they were aware that it did not really fit clearly with government policy in practice, or with 
‘playing safe’ with Ofsted. They cited the reason for this uncertainty with how it would be perceived as 
being due to a lack of empirical evidence supporting the PBL type of approach. However, this is also 
the case with a lot of educational approaches e.g. what works for white working class boys performing 
below their peers. 


The school benefited from good community links and at that time teachers were organised on a 
faculty basis, rather than along strict subject lines — both of these factors helped with the organisation 
and implementation. Whilst PBL was a high profile project, the school sought to play this down 
internally so as to reduce the chance of some element of resentment. The PBL teachers were 
organised on a small team basis, with most of their teaching being PBL. Overall the school sought to 
‘grow’ the teachers in to their role, and had great success with new or NQT teachers in the PBL team 
who were able to get support from the lead staff and the PBL processes. Although much of this was 
likely down to the skills and abilities of the PBL lead, it shows the potential value of team working for 
staff support and development. 


For Year 7 about 50% of the timetable was assigned to PBL and this reduced in Year 8. PBL included 
ICT, Science, Design and Technology and humanities, as well as an element of English. The PBL 
team did not however possess the whole range of subject expertise and where necessary the specific 
departments and teachers from elsewhere in the school contributed to the planning and design of the 
REAL Projects and materials. The team were very cohesive which helped with joint planning, and this 
was helped by having a period when the full team could meet each week on Friday afternoons. 


Training and Innovation Unit support 


Two of the teachers and one of the SLT attended the QUEST training in London. This was found to be 
very helpful and in particular helped the school to identify ways in which their educational aspirations 
could be realised as well as training them to effectively implement REAL Projects. The cost of 
attending this, both time and expenses, were described by the headteacher as significant ‘front 
loaded’ costs which it was not possible to continue funding beyond the first year of the project (despite 
continuing with PBL). Therefore, the following year the QUEST training was done in the school by the 
Classroom Coach and the PBL lead. This approach helped them to tailor the project and training more 
to local needs although while still retaining the essential elements of PBL. 


The input of both the Leadership and Classroom Coach was very much valued by the school and PBL 
team. Both worked mainly with the PBL lead as the role of the Leadership Coach, which was normally 
to support the head to establish the right structure and conditions for PBL to work, was not really 
necessary given what the school was already doing. Moving forwards, the school would be happy to 
purchase support as required, but not necessarily the whole package. 


At quite an early stage the school were encouraged by the Innovation Unit to help other schools adopt 
the programme given their positive experience. However, they took a cautious approach to this, both 
through recognising that particular expertise of the Classroom Coaches would not be that easy to 
replicate, and also risk of not paying enough attention to their own needs. The PBL lead has since 
gone on to do some support / consultancy work with another school, but with still being able to support 
the project in her own school, at home, and has benefited professionally from this. 


Perceived benefits and pupil progress 


For the school the most important tangible benefit was improved engagement in the learning process, 
for both pupils and their parents. With regards to parents we were told that far more attended parents 
evenings than previously as well as accessing on-line reports. 
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It will take time before any academic impacts can be properly assigned to PBL, however the school 
feel that by improving engagement even in just one of the target groups (white working class boys) it 
is likely to improve their academic performance. The school also pointed to future learning benefits, 
including life-long learning and 21° century skills. They feel that embedding these skills at this early 
stage should have future benefits, including for pupils going on to do IBs or more vocational courses. 
We were told of improvements in terms of PBL directly supporting the primary / secondary transition 
as well as the longer term integration of the Year 7 pupils in to the school. 


We were also told that the programme is likely to go on to help support local employment needs, for 
example in the area of STEM and progression to Higher Education. It was also evident that the 
programme was playing a valuable role in terms of staff training and support, in particular through 
team working, and more generally, for the school to help meet its broad educational aims. 


Conclusion 


Even though the school was well prepared to implement the project, in their own words, they still went 
through a ‘massive learning curve’ in implementing the project. PBL did provide many of the benefits 
they sought as well as a number of further benefits. They did, however, point to the quite substantial 
costs, both financially and in terms of staff time, necessary to properly implement the programme, but 
felt this to be a good investment in future teaching and learning. 
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Appendix F: Security classification of trial findings 


5a 


4a 


38 


Design 


Fair and clear quasi- 
experimental design for 
comparison (e.g. RDD) with 
appropriate analysis, or 
experimental design with 
minor concerns about validity 


Well-matched comparison 
(using propensity score 
matching, or similar) or 
experimental design with 
moderate concerns about 
validity 


Power 


Attrition* 


MDES <0.2 


MDES < 0.4 


0-10% 


11-20% 


21-30% 


28 


Weakly matched comparison or 
experimental design with major 
flaws 


MDES <0.5 


18 


oa 


Comparison group with poor or 
no matching (E.g. volunteer 
versus others) 


No comparator 


MDES < 0.6 


MDES > 0.6 


41-50% 


> 50% 


Adjustment 
for Balance 


[-1] 


Adjustment 


forthreats |_____ 


to internal 
validity 


ratings (see notes below) 
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Reason for adjustment for threats to validity (if made): none 


Initial padlock score: lowest of the three ratings for design, power and attrition = 2 padlocks 
Reason for adjustment for balance (if made): differential attrition and imbalance in Ofsted 


Final padlock score: initial score adjusted for balance and internal validity = 1 padlock 
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